I’m Sowmya Jayaram Iyer

A translation medium for text/speech to Indian Sign Language is built using supervised learning. From ISL educational videos, human 2D pose is estimated. This project tackles problems in pose data such as missing joint information and occlusions. Later, 2D pose data is transformed into 3D pose data using a fully connected depth-regressor model to obtain the depth information for each frame. The architecture has a consistent performance rate, for handling missing/occluded joints and obtaining depth in the output, in the GPU as well as CPU processor. This creates the first Indian Sign Language motion capture database. For the translation medium, nltk library is used to grammatically change input text in English to ISL (order of parts of sentence/ tense). Output 3D data stored as .bvh file is rendered on Blender to give action ISL output.

Bayesian-Models are used to defend ML-Models from gradient-based adversarial attacks in images. Adversarial (perturbed) images have the ability to reduce a ML-model's performance by forcing mispredictions. BNNs take both model and data uncertainty into account which gives them a probabilistic robustness against adversarial attacks. Using Variational Inference and Local Parametrization Tricks, Bayesian AlexNet (BAlexNet), Bayesian 3conv3fc (Simple CNN) and Bayesian LeNet(BLeNet) outperform their baseline counterparts. MNIST and CIFAR10 datasets are made adversial using both large epsilon as well as small epsilon perturbations (strength of attack) using FGSM, BIM and PGD attacks. Bayesian models increase the accuracy in prediction of the baseline models by atleast 60% against adversarial images reducing misclassification.

Conditional Generation for Text using T5 (text-to-text) transformer was used to generate a creative title based on context for the given Movie/ TV show description (1024 tokens). A dataset compiling Netflix, Amazon Prime, Hulu and Disney Plus Movies/ TV shows was created to fine-tune the conditional generative transformer. The model achieved a rougeLsum score of 24.9. The fine tuned model has been deployed in HuggingFace as a package and can be directly importd as a pre-trained model.

Implemented image classification using VGG16 and Resnet18 (FastAI) on a cluttered ISL alphabet image dataset comprising different skin colors, backgrounds, and lighting conditions of ISL alphabets. Achieved 93.9% accuracy using VGG16 and 91.8% accuracy using Resnet18. Used this result to build a video-based ISL-spelling to text translation application which can bridge communication between the differently abled people with hearing difficulty and the abled.

Network traffic is classified using deep packet inspection in pcap traces. Packet information (source_ip, dest_ip, etc.) is used to predict QoS Labels for Network Classification. Data is obtained from MAWI Working Group Traffic Archive for a period of three weeks. Using multi classification models- Desicion Tree, Random Forest, Naive Bayes, Logistic Regression, achieved a 99.9% train accuracy and a test accuracy of 94.5%.

Designed and implemented a solution to reduce the effect of cache contention on the performance of network IO and demonstrated our results. Code sections that access the cache implemented are partitioned and isolated in Intel RDT for cache-partitioning and Unikraft Unikernel library. This design overcomes traditional system drawback of heavily CPU or memory bound applications not benefiting much from DDIO tuning based on cache partitioning as observed in ML and DL. Throughput is increased for most benchmarks when going from OS to UniKernels due to the smaller memory footprint and less cache contention.

A U-Net segmentation model (CNN) for accurate segmentation of blood vessels in retinal images used to identify various stages of Diabetic Retinopathy. U-Net allows for more paths for information flow and can be viewed as an ensemble of FCNs. On DRIVE and CHASE DB1 datasets, the model achieves a AUC-ROC of 0.98127 surpassing other state-of-the-art deep-learning based methods.

This project presents a comprehensive survey of Transformer techniques specifically geared towards multimodal data up until 2022. The contributions of this survey include giving:

(1) a theoretical review of Multimodal learning, Transformers, Vision Transformers, and Multimodal Transformers,

(2) a review of multimodal Transformers through the perspective of two important applications- specific multimodal tasks,

(3) to summarize commonalities in challenges faced and designs of existing Transformer models.

Unsupervised hierarchical clustering model predicts delivery driver behavior in B2C companies that provide electric-vehicle based delivery service. Analyzed and classified stops as hubs, driver location, illegal stops, and delivery locations. Data from electric two-wheelers with GPS location firing every few seconds was reinforced with location descriptions extracted using Google Places API and clustered as mentioned labels.

Web Development Project which lists suggests users relevant courses based on their career paths. Scrapes courses from existing MOOCs platforms based on subject/career preferences using keyword matching and enables users to compare ratings and prices across different websites.