Category: Uncategorized

Posted on

Summer Interns, 2025!

interns in AGH
Altar, Terry, Guy

Here in the deep snows of winter (okay, one snow, not very deep, but 5 inches is not bad these days), we offer a throwback to summertime, to share the experiences of this year’s summer interns. Mentored by PhD students Yu Feng and Xingyu Fu and postdoc Tomer Wolfson, with input from former postdoc Vivek Gupta, the six students had come to Philadelphia from Arizona, California, China, and Israel to spend some or all of their summer with us, working on their various projects. It was great to have them join our ranks and participate in our research. I’ve asked them to give their thoughts on the experience and what they learned; here are five responses.

interns and mentors in AGH
Jen, Xuan, Prasham, Rohit, Tomer, Terry, Yu

[Some responses were compiled or polished using AI.]

Rohit Khoja
Master’s student at Arizona State University

This summer, I worked on improving retrieval and question answering over both unstructured and tabular data. My main focus was on enhancing retrieval precision and LLM reasoning accuracy on the OTT-QA and STaRK datasets, particularly for questions that require multi-step reasoning. As part of this, I built a GraphRAG system from scratch. Instead of relying on conventional knowledge graphs at the entity level, we constructed a chunk-level graph, where each node represented a chunk of 7–8 sentences and explored relationships between chunks. This approach led to improvements in retrieval quality and reasoning performance.

Beyond the technical work, I really enjoyed my time in Philadelphia, the beautiful summer weather, exploring different places around the city, and especially the collaborative atmosphere in the lab. Prof. Dan, Tomer and Jennifer were supportive and always ready to help, and the convenience of the Penn Transit service around campus was great.

During this internship, I learned a lot about the retrieval field and recent advancements in it, gaining a deeper understanding of why retrieval is such a critical component of modern AI systems. I also learned how to handle large datasets efficiently on GPUs and optimize data processing pipelines for large-scale experiments.

Prasham Titiya
Master’s student at Arizona State University

Prasham at MIT in Cambridge, MA

This summer, I had the wonderful opportunity to be a research intern with the Cognitive Computation Group at the University of Pennsylvania. I worked on developing information retrieval systems for structured, semi-structured, and unstructured data, focusing on how building a knowledge graph can improve retrieval performance and help with more efficient multi-hop reasoning. The project was rewarding, exploring how semantic and lexical relationships can be represented and leveraged to make retrieval more accurate and context-aware.

I learned a lot throughout the project, both technically and personally. Meetings and discussions with Prof. Vivek Gupta, Dr. Tomer Wolfson, and especially Prof. Dan Roth were incredibly insightful and formative. Prof. Roth’s feedback and perspective on approaching this problem helped me think more critically and systematically about my work. I also had the chance to meet PhD students and researchers from other labs, which broadened my horizons and gave me a better understanding of the variety of work happening in this field. I am especially grateful to Jennifer Sheffield for being so proactive and helpful throughout the course of my internship.

Outside of research, I really enjoyed my time in Philadelphia. The city was lively and full of great places to explore. The UPenn campus was very beautiful, filled with greenery, and had a historic charm that made it a wonderful place to spend the summer. Overall, it was an amazing experience and definitely one of the highlights of my year. I had a great time learning, working with incredible people, and exploring a new city.

Tomer, Alon, Terry, Xuan, Yu, Jen at the Penn Park Orchard

Terry Tong
Undergrad at the University of California, Davis

Hi! I’m Terry, I’m a rising senior at UC Davis. Over the summer, I worked at CCG with my mentors Yu Feng and Prof. Dan Roth. When deciding on a project, I wanted to challenge myself to work on research that is more theoretical in nature, settling on a project on theory behind neuro-symbolic integration in reasoning. 

Terry and the Statue of Liberty

I learned a lot with Dan and Yu. Dan is really good at pruning the research idea search space given his decades of experience in the field, which has saved us a lot from dead ends. We had a discussion on the characteristics that made LMs amenable to tool learning, and I vividly remember Dan bringing up ‘teachability of models’ and how people used to research this, but stopped for good reason. We stopped in our footsteps there and pivoted right away. He’d also bring up relevant papers from long ago (like before I was born) that guided the field into what it is now, e.g. one of his seminal ‘Learning to Reason’ papers. This has always helped me gain perspective on what’s important. While Dan is a busy person, whenever we did meet, I always found it helpful to answer some of the ‘big picture’ questions he asked. I felt challenged to step back from whatever low level details I was implementing and critically think about what we should prioritize—which has improved my research decision making skills overall. 

While I periodically met with Dan, I got a lot of help from Yu. My previous mentors have been more hands off, so when Yu would challenge some of my ideas, I actually found I preferred this type of back and forth. She would tease out details perhaps another researcher would ask, flesh out low-level ideas, which really complemented Dan’s style of high-level advising. I used to think research was all really technical math and derivations but actually found that scientific communication was really important, especially when time is limited and you have to pitch an idea, or get feedback on an experiment design. Making sure the other party knows exactly what you’re talking about helps decision making and ultimately the efficiency of the project. 

Personally, this was the first time I got to research full time outside of classes. I’ve always struggled with context switching between research and classes, so it was rewarding to just have a big chunk of time to let ideas flow. I think I nurtured a habit of trying to understanding things deeper, to spend time digging into neat ideas and deriving equations from scratch. It was really cool to reuse things like learning theory, or theory of computation, that I’d glossed over in my undergrad classes thinking I’d never use them again. Most importantly, this gave me more time to develop my research training skills. I’d be able to reflect on what went well during research and just do `film study’ (see Jacob Steinhardt’s blog on this) and become a better researcher. 

I’m grateful to both Yu and Dan for this opportunity, and all the other CCG members who made my time more enjoyable. The outings we would have w/ Jen to the Penn orchard or the Museum, the Coffee runs I’d have with Tomer, and lunches w/ Alon all helped keep me a happy researcher. 

Visiting the Penn Museum of Archaeology and Anthropology

Altar Horowitz
Undergrad at Tel Aviv University

Hey! My name is Altar, and I’m a second-year Bioinformatics student at Tel Aviv University. This summer, I had the privilege of working on a project in Professor Dan Roth’s lab, alongside my incredible research partner, Guy Kouchly.

Our project had two main parts. The first involved building an online tool that allows AI researchers to compare distances between embeddings of different sentences, based on their chosen embedding type. The second part focused on exploring whether prompt enrichment improves AI retrieval performance from a database.

For me, this was a very special experience – it was the first time I built an entire tool from scratch, which, as anyone who’s done this knows, is a truly unique and educational process. Moreover, being part of such a high-level laboratory and creating a tool that can be used by some of the best scientists in the world was incredibly empowering.

Another highlight of the summer was the honor of working with the amazing Dr. Tomer Wolfson, who dedicated so much of his time to advising and helping us. Overall, this was one of the most meaningful experiences I’ve had, and it definitely strengthened my motivation to keep working hard and pursue a path in the academic world!

Guy Kouchly
Undergrad at Ben Gurion University of the Negev

During the summer, I worked together with my research partner, Altar, on developing a demo for comparing text embeddings and visualizing the distances between them under different models. Later on, we joined another project under Tomer’s guidance, focusing on improving retrieval methods for large language models using prompt engineering. We experimented with a subset of questions from the OTT-QA dataset and evaluated GPT’s ability to retrieve the corresponding “gold” documents. Our approach involved generating a fictional document (using GPT) for each gold document and using it as a prompt. While this method didn’t yet improve results, Tomer believes there’s still potential – especially with more challenging datasets.  

I really enjoyed working on these projects this summer. NLP is new to me, and I’m grateful for the chance to gain hands-on experience so early in my studies. Just as importantly, the lab atmosphere was wonderful – I always felt comfortable asking for help, and everyone was incredibly kind, patient, and welcoming.

In terms of what I’ve learned—almost everything was new! On the theoretical side, I got to explore concepts like embeddings, dimensionality reduction (PCA), and retrieval-based reasoning. On the practical side, I learned about building demos, using APIs, and the general workflow of conducting research.

Many thanks to Dan, Tomer, Terry, Rohit, Prasham, and you, Jen, for all the support and for making this such a meaningful experience.

Posted on

Summer Interns, 2024!

This summer, for the first time in a few years, we had a substantial group of summer interns working with us, mentored by PhD student Sihao Chen and postdoc Vivek Gupta. Some were visiting from universities as far away as Utah and California, but the group also included a Penn undergrad and a local high schooler. We were honored to work with them and incorporate them into our research work. As the summer term winds up, I’ve asked for their thoughts on the experience and what they learned and enjoyed here!

Harshavardhan Kalalbandi
Master’s student at the University of California, Riverside

My internship at Prof. Dan Roth’s Cognitive Computation Group has been an incredible journey. During the first few weeks I spent time going through many research papers to formulate a good problem. I found a keen interest in temporal QA on tables, where the tables can be represented in various ways. We started by establishing baselines using existing reasoning methods. We proposed a dynamic reasoning approach, allowing the model to have flexibility in formulating its approach. Initially, we tried solving the problem through simple prompting, but when that proved insufficient, we explored alternative approaches.

Harsha on a trip to Washington, DC

Our efforts led us to develop a multi-agent approach and a fine-tuning method for our system. A significant challenge has been keeping pace with the rapid advancements in state-of-the-art NLP research. A highlight of this experience was meeting Prof. Dan Roth and his incredible team, which was both inspiring and fun. Dr. Vivek Gupta’s mentorship and expertise was very helpful in seeing through this work. I also explored the incredible campus of Penn, went around Philadelphia, and traveled to New York City and DC during the weekends. It was a great fun experience where I learned a lot, met incredible people, and enjoyed myself.

Kushagra Dixit
Master’s student at the University of Utah

This summer, I had the amazing opportunity to intern with the Cognitive Computation Group at the University of Pennsylvania. During my time here, I worked on two fascinating domains: improving the temporal understanding of large language models on semi-structured data and exploring the mechanisms behind in-context learning in LLMs. These projects allowed me to delve deep into some of the most exciting trends in NLP research, providing invaluable experience that I will carry forward in my career.

Kushagra on a trip to Washington, DC

One of the most enjoyable aspects of this internship has been the vibrant and collaborative environment. I’ve had the pleasure of interacting with PhD students in the lab, sharing ideas with fellow interns, and receiving invaluable guidance from Dr. Vivek Gupta. Every discussion with Professor Dan Roth has been particularly special. His insights and expertise have consistently challenged and inspired me, leaving a profound impact on my approach to research. I will leave this internship not only with a wealth of new knowledge but also with cherished memories of the fun and engaging moments shared with the team.

In terms of skills, I’ve significantly sharpened my programming abilities, but more importantly, I’ve learned how to drive a research project from inception to completion. Engaging with so many experienced researchers has provided me with numerous opportunities to understand their thought processes, which has been a critical learning experience. As I move forward, I am excited to continue my engagement with CogComp, building on the strong foundation this internship has provided, and contributing further to NLP research.

Mike Zhou
Undergrad at the University of Pennsylvania

Mike at Mission Peak, San Francisco

Hi! My name is Mike and I’m currently a rising senior at Penn. This summer, I’ve had the pleasure to be a part of the cognitive computation lab, where I’ve been focusing on language model reasoning and understanding. My current projects involve exploring large language models’ abilities to reason, and whether their reasoning comes from a form of deduction or simply from semantic associations. My day-to-day routine in the lab involves keeping up with literature, designing experiments that I’ll run, and talking with other lab mates to give and gain insights on each other’s projects. Overall, I’d say that I’ve learned quite a bit about and gained a good number of insights on language models this summer, whether it was from my work, peers, or Professor Roth.

Yanzhen Shen
Undergrad at the University of Illinois Urbana-Champaign

Hi! I’m Yanzhen Shen, an undergraduate student at the University of Illinois at Urbana-Champaign. This summer, I had the privilege of conducting research at the University of Pennsylvania under Professor Dan Roth and my PhD mentor, Sihao Chen. During this experience, I worked on cutting-edge research in Information Retrieval and gained a deeper understanding of the qualities of a good researcher.

Yanzhen in Chicago

Our project focused on improving retrieval systems, particularly for complex queries. While current systems handle simple queries effectively, they struggle with ones with logical operators, such as “Orchids of Indonesia and Malaysia but not Thailand.” To address this, we are using text embedding models to better interpret logical operators like AND, OR, and NOT in complex queries. Our goal was to push the boundaries of how query and document dense embeddings can represent complex information.

Technically, I became proficient in dual encoder training frameworks and data processing for Information Retrieval tasks. 

More importantly, discussions with Professor Roth helped me view our ideas from broader perspectives. For example, he continued to encourage me to think about how our model differs from existing dense retrieval models and other retrieval systems. In addition, working closely with Sihao also gave me insights into how a senior PhD student approaches and resolves challenges in complex research problems. We also engaged in paper discussions, where he taught me to think about critical questions when reading papers, such as “What makes a good researcher question?” and “ Is this work influential?”

Atharv Kulkarni
Undergrad at the University of Utah

Atharv in conversation with Benjamin Franklin

This summer, I had the incredible opportunity to intern at the University of Pennsylvania’s Cognitive Computation group under the guidance of Professor Dan Roth. The main focus of my internship was to enhance the capabilities of language models for complex reasoning tasks using neurosymbolic approaches. I created a question-answer dataset based on the Wikipedia information of Olympic sport athletes and used it to evaluate state-of-the-art language models. This involved integrating Wikipedia data into a database and developing SQL queries to create a large synthetic dataset. I discovered that models often struggle with temporal reasoning tasks, which led to productive discussions with Professor Roth about using neurosymbolic techniques to improve reasoning performance. By refining models with fine-tuning and prompt engineering, I made significant progress in enhancing language models’ ability to transform natural language questions into SQL queries.

Beyond the technical work, Professor Roth’s mentorship was invaluable. His insightful feedback and guidance helped me develop skills in fine-tuning models and utilizing various APIs, significantly advancing my understanding of the field. His expertise provided me with a deeper appreciation for the nuances of research and inspired me to think critically about complex problems. Mentoring Aadit Bontha, a high school student interning with our CogComp group, was a rewarding experience, offering me a fresh perspective on language models. Exploring Philadelphia’s iconic sites, such as City Hall and the Benjamin Franklin Museum, along with my peers, added to the memorable experiences of this summer. Overall, I gained a deeper understanding of research and thoroughly enjoyed collaborating with my peers. I am grateful to Professor Dan Roth and Dr. Vivek Gupta for this invaluable opportunity.

Aadit Bontha
HS Student at La Salle College High School

During my internship at UPenn, I had the opportunity to work on a project centered around large language models (LLMs) and their ability to extract and organize data from various sources. My main task was to explore how these models could generate structured answers, like tables, from diverse data inputs—a challenge that required both technical skill and a deep understanding of AI.

Mentor Vivek Gupta with Aadit, holding matching program certificates

I started by working with the TANQ dataset, which includes a vast collection of question-answer pairs. My role involved extracting relevant information and using LLMs to generate structured outputs. This required extensive coding in Python, utilizing tools like BeautifulSoup for web scraping and the JSON module for data parsing.

One of the most interesting aspects of the internship was experimenting with different AI models, such as GPT-3.5 and Gemini 1.5. Through this, I gained insights into how these models perform in various scenarios—GPT-3.5 excels at handling complex queries, while Gemini 1.5 is more effective with simpler, structured tasks.

I also focused on refining the prompts we used to optimize model performance. This involved understanding how to craft questions that would elicit the most accurate and relevant responses from the models. Overall, the internship was a highly educational experience that enhanced my skills in AI and data processing, providing me with valuable insights into the practical applications of these technologies.