Browsing by Author "Jiang, Mingchao"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Generative Language Models for Program Synthesis and Evaluation
    (2024-12-06) Jiang, Mingchao; Jermaine, Christopher M.
    Recent advances in Large Language Models (LLMs), such as GPT and Claude, have significantly advanced the field of program synthesis. To evaluate the performance of these models, traditional benchmarks like APPS, MBPP, and HumanEval reveal limitations due to potential data leakage and their inability to mirror the complexity of real-world programming. These benchmarks typically feature concise, stand-alone code samples that fail to assess the nuanced capabilities required for comprehensive coding tasks adequately. To address these limitations, this dissertation introduces a novel, private benchmark dataset - SimCoPilot, specifically crafted to simulate the ability of an AI such as a large language model (LLM) to perform as a “copilot”-style, interactive coding assistant. In SimCoPilot, an AI is asked to provide small amounts of code within an existing project, ranging in size from hundreds to thousands of lines. The benchmark tests an AI’s ability to write code in both completion (providing code to finish a method or a block) and infill scenarios (providing code to fill a blank in a method), covering various domains such as classic algorithms, databases, computer vision, and neural networks. Despite their varied architectures, most LLMs typically treat source code as mere string objects and require large-scale models and extensive training datasets. Unlike natural language, however, source code is a formal language imbued with rich syntactical and semantic structures. Addressing this disparity, this dissertation explored an innovative approach that explicitly extracts and integrates these syntactic and semantic elements into an encoder-decoder transformer model. Our detailed evaluation analyzes how LLMs manage different code dependencies and logic complexities, providing insights into their operational effectiveness in realistic programming environments. This examination provides profound insights into the capabilities of modern Language Models in navigating realistic programming challenges, thereby making a significant contribution to the understanding of their practical applicability in the software development environment.
  • Thumbnail Image
    Item
    Influence based bit-quantization for machine learning: Cost Quality Tradeoffs
    (2020-10-30) Jiang, Mingchao; Palem, Krishna V.
    Due to the significant computational cost associated with machine learning architectures such as neural networks or network for short, there has been significant interest in quantizing or reducing the number of bits used. Current quantization approaches treat all of the network parameters equally by allocating the same bit width budget to all of them. In this work we are proposing a quantization approach which allocates bit budgets to parameters preferentially based on their influence. Here, our notion of influence is inspired by the traditional definition of this concept from the Fourier analysis of Boolean functions. We show that guiding investment of bit budgets using influence can get acceptable accuracy with lower overall bit budgets when compared to approaches that do not use quantization. We show that by trading 4.5% in accuracy, we can gain in bit budgets by a factor of 28. To better understand our approach, we also considered allocating bit budgets through random allocations and found that an our influence based approach outperforms most of the time by noticeable margins. All of these results are based on the MNIST data set and our algorithm for computing influence is based on a simple and easy to implement greedy approach.