Category: today i learned (til)

The creation of language models like ChatGPT is rooted in the field of artificial intelligence and its subfield of machine learning. Specifically, ChatGPT belongs to the family of models known as transformers, which were introduced in a seminal 2017 paper by Vaswani et al. titled “Attention Is All You Need” [1].

Before we delve into transformers and attention, let’s talk about the more basic types of machine learning models: supervised and unsupervised learning.

Supervised learning involves training a model to make predictions based on labeled data. For example, you might train a model to recognize handwritten digits by giving it many examples of handwritten digits along with their correct labels (e.g., “this is a 7,” “this is a 9,” etc.). The model then learns to recognize patterns in the data that correspond to the correct labels, and can make predictions on new, unlabeled data.

Unsupervised learning, on the other hand, involves training a model to find patterns in data without explicit labels. For example, you might train a model to group similar news articles together based on their content, without telling the model which articles belong in which groups. The model would learn to identify common themes or topics across the articles and group them accordingly.

Now, let’s talk about transformers. Transformers are a type of neural network that were introduced in the aforementioned 2017 paper [1]. They’re used for a variety of natural language processing (NLP) tasks, such as language translation, text classification, and text generation.

The basic idea behind transformers is that they use “self-attention” to process input sequences. Self-attention means that the model can pay different amounts of attention to different parts of the input sequence, depending on which parts are most relevant to the task at hand [2].

To illustrate this, let’s consider an example sentence: “The cat sat on the mat.” If we wanted to use a traditional neural network to process this sentence, we might represent each word as a vector with a 1 in the position corresponding to the word, and 0s elsewhere and feed those vectors into the network one at a time.

But with a transformer, we represent each word as a vector that includes information about all the other words in the sentence. Specifically, for each word in the sentence, we compute a “query,” a “key,” and a “value” vector. The query vector is used to determine how much attention to pay to each of the key vectors, and the value vectors are combined based on those attention weights to produce a final output vector.

In the case of our example sentence, the transformer might pay a lot of attention to the word “cat,” since it’s the subject of the sentence, and less attention to the word “the,” since it’s just an article. By using self-attention in this way, the transformer can process input sequences much more efficiently than traditional neural networks, since it can focus on the most important parts of the sequence at each step.

Now, let’s talk about how ChatGPT was specifically trained. ChatGPT is based on the GPT-3 architecture, which is a highly advanced transformer model with 175 billion parameters (i.e., weights that the model has learned during training). That’s a lot of parameters – for context, the previous state-of-the-art model, GPT-2, had “only” 1.5 billion parameters.

To train a model as large and complex as GPT-3, you need a lot of data. In fact, GPT-3 was trained on a dataset of over 45 terabytes of text, which included everything from web pages to books to Wikipedia articles. This huge amount of data allowed the model to learn a vast amount of knowledge about the structure and nuances of language.

The training process for GPT-3 involves a technique called gradient descent. Essentially, the model starts with randomly initialized weights, and then makes predictions on the training data. The difference between the model’s predictions and the correct answers (known as the “loss”) is then used to adjust the weights slightly in the direction that would reduce the loss. This process is repeated many times, gradually fine-tuning the model’s weights to better fit the data.

One challenge with training models as large as GPT-3 is that it requires a lot of computational power. GPT-3 was trained using a combination of both CPU and GPU computing, and the training process took several months to complete.

So, what makes ChatGPT different from GPT-3? As I mentioned earlier, ChatGPT is a variant of the GPT-3 model that has been fine-tuned for conversational applications. This fine-tuning process involved training the model on a specific task – in this case, generating responses to user inputs in a natural-sounding way.

To fine-tune the model, the researchers behind ChatGPT used a technique called transfer learning. Transfer learning involves taking a pre-trained model (like GPT-3) and re-training it on a new, related task (like generating conversational responses). The idea is that the model has already learned a lot of useful knowledge from its pre-training, and can apply that knowledge to the new task with less additional training.

During the fine-tuning process, the researchers fed the model pairs of user inputs and responses from a large corpus of conversational data. The model then learned to generate responses that were both grammatically correct and semantically relevant to the user input.

One interesting aspect of ChatGPT is the potential ethical implications of their use. As AI becomes more advanced and capable of generating human-like language, it raises questions about the role of AI in society, and how we should regulate its use. For example, should AI-generated text be subject to the same standards of truthfulness and accuracy as human-generated text? Should we be concerned about the potential for AI to spread misinformation or propaganda? These are complex and important questions that will require careful consideration and debate in the years to come.

In conclusion, the science behind ChatGPT is rooted in the fields of artificial intelligence and machine learning, and specifically in the use of transformers and self-attention mechanisms for natural language processing. ChatGPT was trained on a vast amount of data using gradient descent, and was fine-tuned for conversational applications using transfer learning. As AI continues to advance, it’s likely that models like ChatGPT will become even more sophisticated and capable, raising important ethical and societal questions along the way.

Works Cited

1. Vaswani, Ashish, et al. “Attention Is All You Need.” ArXiv.org, 6 Dec. 2017, https://arxiv.org/abs/1706.03762.
2. Cristina, Stefania. “The Transformer Attention Mechanism.” MachineLearningMastery.com, University of Malta, 5 Jan. 2023, https://machinelearningmastery.com/the-transformer-attention-mechanism/

Howdy! In planning for our event, The Devil Wears Science, in the Atlanta Science Festival, I took some initiative in a few spaces. I have helped to look into the best areas, in and around campus to host our event (and what fits a small budget), with the ultimate answer landing to an on-campus event at our Living Science Building of Kendeda! I’ve also looked into potential partners, and felt impressed with Dr. Jayaraman, a professor here at Georgia Tech in our Material Science/Engineering department and brought him on board. He has been incredible, and invited an impressive colleague, Dr. Sungmee Park to join the effort. Yet, one creative aspect I’ve worked on for this event was very creatively fun and encouraged me to do some thinking on marketing/interactions between our audience! For this, I have been working on some interesting designs and flyers! Starting in fall, I made a general logo that I thought would be fun to put on basically everything. It could be used for flyers, social media, buttons, and more — maybe even given out on some items to commemorate the event! Here, I wanted to discuss the evolution of my design work! Though it’s a small turn from the usual science content, I hope you enjoy.

The picture above shows the general design I made during fall of 2022 – a centerpiece for future event illustrations. First, I wanted to focus on the general color palette that matches the theme. Wanting to stick to the hellish theme, I initially knew black and reds would be crucial, but it took some time to figure it all out. In trying to find how best to represent the devil here, I came to the conclusion that a men’s silhouette would be best, and finding a suave man in a suit was very fitting. Because of this, even though black is dark and brooding, it would not work for the background as it would remove some of the suit details. Instead, I chose a grey to be the background color as its one of my favorite colors and goes well with black and dark reds. Initially, I was actually working with some red dresses and devillish faces, but I enjoyed the pivot of the formal ‘Man In The Suit’ better. Moreover, I found a cool satanic-looking goat head pattern and wanted to include this in the design. Overlaying this on the head leaned into ‘The Devil Within’ vibe that sparked a lot of intrigue. With that solidified, I went on to include more science aspects into the design. I think my favorite inclusions are the subtle aspects such as having the devilish man have a chemical structure handkerchief and holding a small magnifying glass. I feel like it shows that he is interested in science, and is also slightly ‘wearing’ it as well! At the event, I enjoyed that one of the STEMComm faculty members and professor here in the Chemistry Department at Georgia Tech, Dr. Evans, came to the show with some chemical structures clipped onto his blazer! It’s a fun little parallel to this, and even though I don’t think he intended to be similar to the devil man here, it was hard not to think of it!

Onto the background details, I wanted to balance the image with more elements of science on the sides. When I found these detailed images of a microscope and telescope, I knew I wanted to include them somehow. They matched well with the sudden increase in artistic complexity of the goat skull, and added some focus on the STEM interests of the club. In order to tie it in to the design, I used these tridents for the instruments to ‘rest’ on, rather than as a devil tail or something similar. From a physics perspective, I liked the symmetry and the tridents hold a resemblance to the psi symbol (ψ) as it usually represents wave functions used in modern physics and quantum mechanics. The symmetry also reminds me of bra-ket notation also used in these fields, especially when wave functions are involved. Though most people likely won’t think of that, it was something that made me happy. With many scientific representations on and around the Devil, I felt like I was really satisfied with the general design, and it was extremely fulfilling to see it being shared on social media and on the program that was handed out to everyone at the show.

 

With the general design finalized, it was time to apply this to marketing the event. Since I created the design, I took initiative to also create the flyers for the event, which took many iterations to perfect.

 

After reviewing many other flyers I wanted to keep it simple and ensure it included the basic details: event name, short description, location, date, price of tickets + how to buy them, and of course, the design. For the event name, I wanted it to look extremely similar to the flyers for “The Devil Wears Prada” since we’re doing a word play on that, so I found a similar font called RoxboroughCF on Canva. I also created a QR code so that people could easily access the ticketing website, and emphasized the date and location so that people could easily schedule to come to the event! I wanted to have a relatively balanced look, offsetting the teaser + info and ticketing information on either side.

 

Here, I felt like the information didn’t say everything we wanted to say, so the main change I made was taking the information from the ticketing website and copying it onto the flyer. I also offset the ticketing information to balance around the character. Due to the large amount of text, I minimized the size of the devil design, which made some details a little difficult to see. I also really liked having the location and time bolded on the bottom, mirror the symmetry of the tridents on the bottom. I felt like it is still in focus enough that it can’t be forgotten, but not so awkwardly on the flyer. Plus, it allowed for more information about the event.

 

From here, I wanted to cut down on the large wall of text and emphasize the design, but it didn’t feel quite right yet. At the least, there’s more room to show the design!

 

I realized that the ‘text logo’ (vs. the design which I will now refer to as the logo) was too attached to the top and needed margins to look better. I think this is one of the best improvements I had made. Additionally, to help justify the price to others upon first seeing the flyer, I mentioned that we will be serving alcohol and food. The next iteration is the same except I took out ‘for all’ because it seemed redundant and not entirely true due to the legal aspects of alcohol, even if I did declare that it is for those aged 21 and over.

The update is shown above, with considerations in mind! Yet, it still felt word-y, and some of my peers and designer friends agreed. Thus, the final result comes along:

So, I really cut down and emphasized the most important aspects of the event information. Additionally, we wanted to convert the ‘voting’ to focus less on the presentations and instead on audience engagement and hopefully see a lot of fun science themed outfits (which we did)! This way, there was no inequality between presenters that we were extremely grateful to for their help with the event.

With more balance between design and text, this became the final edit with important information bolded for easier reading. Throughout the process, I looked for input from other members of STEMComm as well as from a few friends/co-workers that work on making social media graphics, and we all found consensus on this edit.

Then, for marketing purposes, we realized the importance of having a version of this flyer for social media. We knew that it would be redundant to have the QR code (those holding their phone aren’t going to need to scan with the phone their holding), but we can include links instead! For now, I have two versions that are almost identical, where one includes a tinyurl link just in case social media links get a little busy. We planned to include a linktree in the bios to access where the QR code went to, but I know sometimes on instagram, you have to go through different unnecessary steps that make people less likely to go to the link. Given that this is for social media, I also looked into the ideal aspect ratio/pixel widths so that it looks best on instagram stories or other postings.

Here is first social media flyer iteration:

Then, its evolution:

So, which one is your favorite? Do you think the final versions are the best, or is something not quite right to you? Hopefully, you’ll have already seen the final versions before and it encouraged you to come to the event.

With these finalized, the members of STEMComm all went to social media to share these online and post about the event. We also looked into sharing flyers around campus to encourage more people to come.

Afterward, I thought it would be nice to have small giveaways for everyone that attended. Thanks to the Invention Studio on campus (follow them @gtinventionstudio on instagram!), I knew I could make pins/buttons and more. I didn’t want the full design to be a sticker/button because its large, so I thought to emphasize the details. So, I made separate images including just the smaller elements and had two versions with white and black backgrounds for more options. I ended up with 7 main designs that I created into pins, and I think they turned out amazing and I am so glad we were able to give them all away to the attendees! Thanks to Jalen, we were able to make some stickers as well. Additionally, Dr. Leavey gave us acrylic to work with, so Isabelle and I went to the Invention Studio together and she worked with the laser cutter for some cool devil charms. One of the other STEMComm members, Michelle Stanek, was also able to make earrings from the charms as well, which turned out amazing!

And this was the evolution of the design created for The Devil Wears Science event. I was extremely grateful to use some creative abilities to help the show, along with my conversations with the MSE department leading to the presentations from Dr. Jayaraman and Dr. Park’s contribution. I cannot wait to see what STEMComm ends up doing next year. Thank you to everyone that showed up for this event!

Stay curious, everyone!

Intro/Background

Vegan- I am sure by now you have heard of the term either from someone you know trying it out, or the millions of social media profiles promoting the lifestyle. Whether you have a friend that’s been vegan for years or want to try it out for yourself, it’s not hard to notice the upsurge in popularity it has been gaining for years. I remember growing up pretty traditionally, eating meat and dairy, without a second thought as to the ethics or implications. Simply put, most children follow what parents show them in terms of eating patterns and habits while growing up and don’t think to question it until living independently (excluding picky eaters and young innovative thinkers). My first exposure to veganism was in my own home- a couple of years ago, my older brother had been working at a pizza place for a couple of months and had developed some pretty stubborn acne (the restaurant let the workers eat the leftovers…you can see where this goes). Of course, the dermatologist that he visited told him to analyze his eating habits and try his best to limit dairy and greasy foods; pizza had to go. That’s when he made the extreme turnaround in his eating and decided to go full-on vegan. He quit his job at the pizza place and got a new job at a vegan restaurant to learn how to make dishes. It was an arduous process for him through and through, and although he isn’t strictly vegan anymore, he still struggles with eating “regular” food without guilt-free.

Veganism Explained

Veganism is a word that confuses many because of how simple yet complex its definition is. Veganism purely means a diet without any animal products. Now, this is where it gets a bit hairy- a lot of the foods that people eat on a regular basis have added ingredients that most are unaware of. When I went vegan, it was ASTOUNDING to find out just how little people know about how food is prepared. For example, one of my coworkers last year tried to offer me an M&M and I declined because I was vegan; I told him “I can’t have any dairy”, to which he responded, “M&M’s don’t have dairy!!!”. I was shocked- it was obviously chocolate candy, specifically MILK chocolate. I wondered how people could think like this, blatantly ignorant of what they’re eating. But back to veganism: animal products are a large genre of food that include any kind of animal meat, dairy(milk, cheese, whey), and any kind of animal byproduct like eggs. In some cases, vegans will not eat gelatin because it is created from animal bones. In cases where veganism goes from a diet to a lifestyle, vegans will not wear any clothes or use any products that come from animals, like leather.

Environmental Perceptions

The story that follows veganism is that it “will cure the earth and society’s morals”, meaning if nobody is eating or using animals for products, then the world would have more animals to sustain the lands and ecosystem. As well as, abstaining from killing animals for food would better the morals and ethics of society as more cohesive and united to the earth. Moreover, it is no secret that the meat industry is one of the leading causes of high emissions of greenhouse gases, one of the main contributors to climate change. The animals that we discuss in regard to the food industry are cattle, “who are estimated to cause approximately half of the food-related climate emissions due to land use impacts and methane produced by ruminating animals”(Kortetmäki, Oksanen 2021). Knowing this and pairing it with the veganism diet itself makes it clear to see that it is commonly held that going vegan would greatly benefit the environment. Joseph Poore, a writer for the journal Science, once stated in an article regarding veganism that “a vegan diet is probably the single biggest way to reduce your impact on planet Earth, not just greenhouse gases, but global acidification, eutrophication, land use and water use”. It’s almost as if by not going vegan, you are adding to the climate catastrophe that the world is facing and not doing your part.

What Researchers Say about the Environmental Impact

Before getting into what the scientific community has been reporting about the true impact of veganism versus traditional diets, I wanted to clarify why it is veganism that is being highlighted as a solution to the climate crisis instead of regular vegetarianism. Vegetarians do not eat any kind of meat, but they can and do enjoy all types of dairy and eggs. Cheese, one of the world’s most beloved foods, has “one of the highest GHG footprints among foods and is common in non-meat diets” (Kortetmäki, Oksanen 2021). Therefore, veganism is a better alternative to vegetarianism when discussing solutions that would most benefit the climate.

So, what are researchers really saying about veganism and the environment? An article that analyzed the argument for climate-centered veganism brought up three big points worth discussing. The first point and probably the most obvious, was that there are animal-based foods that have close to or even lower carbon emission rates than some plant-based foods. For example, “emissions of a small, 100-g piece of chicken (approximately 1.15 kg CO₂eq) equals the emissions of approximately 1.1 L of soymilk or 500 g of tomatoes” (Kortetmäki, Oksanen 2021). That’s crazy! To think that chicken and soy milk have similar carbon emissions is mind-blowing. The second point that was introduced was the food waste problem. A vegan person that is accidentally served chicken ramen instead of veggie ramen would have to send back the original, thereby wasting a nutrient-dense and high-carbon emitting food. This is just one example but can be applied to the grander scheme of things. Lastly, and most important in my opinion, veganism and avoidance of animal-based foods advance the climate-friendliness of a diet, BUT do not guarantee it! The reason behind this is that foods with high fat and protein content have higher carbon footprints than foods that are primarily carbohydrates. This means that foods like avocados, chickpeas, and tofu still have high carbon footprints, and therefore are still contributing somehow to climate change. Thus, a vegan person eating only fast food would not be benefiting the environment whatsoever

Conclusion

It is hard to say for sure at this point if veganism is strictly bad or strictly good for the environment, but we have some clues as to which direction the pendulum is currently swinging to. Although the research outlined above does a great job of poking holes in the climate-based argument for veganism, I still find that the environmental pros outweigh the cons. A lot more research is definitely needed to come to a real conclusion on this topic, and I am excited to see what comes of future findings. For now, I stick with the fact that veganism is more beneficial to the environment than an animal-based diet.

 

References

Kortetmäki, T., Oksanen, M. Is there a convincing case for climate veganism?. Agric Hum Values 38, 729–740 (2021). https://doi.org/10.1007/s10460-020-10182-x

You’re wandering through the forest – lost, worried, thirsty. Your phone is at 6%. You have AT&T, so of course you have no service. You’re a 21 year old raised in the suburbs of Chicago. You’ve barely seen a tree, let alone have any survival skills. 

When all of a sudden, your phone buzzes with a YouTube notification. Your phone just connected to a 5G tower that was disguised as a tree. You call 911. You’re saved. Thank you so much 5G!! Yet, once you return to society, you don’t return the favor to 5G; you just continue to live clueless about what 5G is or how the thing that just saved your life works. This is, of course, an extremely relatable story. We’ve all been there. So I’m here to fulfill your life debt to 5G and explain how it works.

5G is the latest technology for mobile communication networks that promises faster internet speeds, lower latency, and greater capacity to connect more devices simultaneously.

To put it simply, you know how your phone connects to the internet using cellular data or Wi-Fi? 5G is the next and latest version of that connection, and it’s better than previous versions. It’s like upgrading from an old bike to a faster one.

With 5G, you’ll be able to download movies, music, and other files much faster than you do now. In fact, you’ll be able to download a full-length HD movie in just a few seconds. That’s like getting your candy bar in less than a second at the checkout counter!

5G is a type of cellular network where the service area is divided into geographic cells. All devices with the antenna that allow them to connect to 5G wireless access the radio waves generated from these towers.  The towers also have antennas to pick up on 5G signals sent back from the devices. These radio waves transmit and accept information.

The technology behind 5G is also more efficient, which means that it won’t drain your phone’s battery as quickly as older networks. It’s like having a phone that lasts longer before you need to charge it.

There is also something known as “beamforming” used in 5G networks which is worth discussing in this overview. Beamforming is a way of processing signals for directional signal transmission/reception. As you can see, different antenna quantities can affect the efficiency and range of signal processing. Beamforming can be divided into two categories: digital and analog. Digital beamforming involves the transmission of data across multiple channels, while analog beamforming focuses on shaping the radio waves to be directed in a particular direction. 

You now understand the very basics of 5G and have fulfilled your life debt. Congrats!

—

Hoffman, Chris (January 7, 2019). “What is 5G, and how fast will it be?“. How-To Geek website. How-To Geek LLC. Archived from the original on January 24, 2019. Retrieved January 23, 2019.

Yibo, Wang. “What Is Beamforming?”. Info-Finder. Huawei Technologies Co., Ltd. Updated February, 4, 2022. Retrieved February 7, 2023.