Published using Google Docs
cosas interesantes
Updated automatically every 5 minutes

cosas interesantes

Para que no se olviden:

생각想法pensamientosthoughts/theories

God of the Gaps

Fermi Paradox

“What-The-Hell” Effect

Zipf’s Law

Pareto Principle

Infinite Monkey Theorem

100 Prisoners Riddle

Trolley Problem

The Chinese Room

Mary’s Room

Brain in a Vat

Friendship Paradox

Black Swan Theory

Allais Paradox

Mandela Effect

Semantic Satiation

The Linguistics Iceberg Explained

Dangers of population growth facilitation

Fundamental question of rationality - “Why do you believe what you believe?”

Reciprocity Bias

Suspension of disbelief

Lateral thinking

The Cobra Effect

Heisenberg Effect

Dunning-Kruger Effect

Straw man fallacy

Sapir-Worf hypothesis

Unexpected hanging paradox

Fallacy fallacy

Pro-drop language

A) 这块蛋糕很好吃。 谁烤的？

B) 不知道。喜欢吗？(I) don’t know. (you) like (it)?

Escher Sentence

Pandas suck at existing

The Savanna Hypothesis (Pliocene)

Lack of Terrestrial Bioluminescence

Poe’s Law

Big-Headed Ants’ Rippling Effect

Portmanteau

Newcomb’s Paradox

Chinese dialects

책书librosbooks

“죽고 싶지만 떡볶이는 먹고 싶어”

“Blink” by Malcolm Gladwell

“The Man Who Mistook His Wife for a Hat and other Clinical Tales” by Oliver Sacks

“The Confidence Game” by Maria Konnikova

“The Tipping Point” by Malcolm Gladwell

“Sapiens: A Brief History of Humankind” by Yuval Noah Harari

“Homo Deus” by Yuval Noah Harari

“The Three-Body Problem” by Cixin Liu

“Algospeak” by Adam Aleksic

  1. Social media platforms make experience as good as possible for users to build up following
  2. Once users are locked in, platforms will make experience was good as possible for businesses trying to advertise to the users
  3. Once users and business are locked in, platforms will exploit them to make money

HARRY POTTER 해리 포터

“Harry Potter y la piedra filosofal”

“Harry Potter y la cámara secreta”

“Harry Potter y el prisionero de Azkaban”

“Harry Potter y el cáliz de fuego”

“Harry Potter y la Orden del Fénix”

“Harry Potter y el misterio del príncipe”

“Harry Potter y las reliquias de la muerte”

역사历史historiahistory

Siege of Numantia

Eyam

동물动物animalesanimals

Cymothoa exigua

Leucochloridium paradoxum

cosas de informatica

SQL vs NoSQL Databases

PostgreSQL vs MySQL

Relational databases

DevOps

SSL Certificate

Deep Learning - “Deep Learning: Foundations and Concepts” by Christopher Bishop and Hugh Bishop

Deploying Flask Backend (https://www.animales.click)

Getting SSL certificate on yangba.net

CPU vs GPU

Process vs Thread

Firestore

OAuth 2.0 (Open Authorization 2.0)

  1. Initiate login on Github
  1. Github = client
  1. Github redirects browser to Google
  1. Github constructs special url with client_id, redirect_uri, scope, etc.
  1. User authenticates and grants consent on Google
  1. Enters username and password
  2. Google displays consent screen: “Github wants to access…”
  1. Google sends authorization code back to Github
  1. Google’s authorization server redirects back to Github’s specified redirect_uri, which contains authorization code
  1. Github exchanges authorization code for access token
  2. Google issues access token
  1. Validates authorization code and sends response back that contains access token
  1. Github uses access token to access google data
  1. Sends access token in Authorization header of API requests to Google (e.g. Authorization: Bearer <access_token>
  1. Github logs user in
  1. Google’s resource server returns requested user data and Github uses this to log in / create new account

JSON Serialization

base64.b64encode(byte_image).decode(‘utf-8’) //valid for JSON serialization

if __name__ == “__main__”:

Deep Learning Crash Course

Deep Learning Crash Course for Beginners

  1. Gathering Data
  1. IRIS Flower Dataset ~ 150 images
  2. Google Translate ~ trillions of examples
  3. Amount of data ~ 10x # of model parameters
  1. Preprocessing
  1. Split into subsets—train-test-validation splits
  2. More hyperparameters → larger validation set
  3. Cross-Validation
  1. Training
  1. Feed data → forward propagation → loss function → backpropagation
  1. Evaluation
  1. Test model on validation set
  1. Optimizing
  1. Hyperparameter Tuning e.g. increase # of epochs adjust learning rate
  2. Regularization, data augmentation (more data)

Multicollinearity

pd.get_dummies() = ONE-HOT ENCODING

K-fold Cross Validation

Tuning vs Fine-Tuning

CS148 Project ML

Linear regression vs logistic regression

Scikit-learn methods

Joblib

XGBoost (eXtreme Gradient Boosting)

swar/nba_api timeout error

cosas de llms

Hands-On LLMs

Ch1: An Intro to LLMs

Ch2: Tokens and Embeddings

  1. Tokenization method
  1. Word, subword, character, byte
  2. GPT-2 tokenizer represents special characters (e.g. Chinese characters) by multiple tokens (all look identical i.e. question mark black box) but stand for different tokens
  3. GPT-4 tokenizer has specific token for every sequence of whitespaces up to 83
  1. Tokenizer design (parameters)
  1. Vocabulary size (e.g. 100K)
  2. Special tokens
  3. Capitalization
  1. Dataset to be trained on
  1. E.g. code-focused models more optimized toward encoding code by making different tokenization choices

Ch3: Looking Inside LLMs

  1. Self-attention layer
  1. Relevance scoring + combining information
  2. Attention mechanism is duplicated and executed multiple times in parallel → each parallel applications of attention is conducted into an attention head
  3. Attention calculation
  1. Goal: produce new vector representation of current position based on previous tokens (vector representations)
  2. 3 projection matrices: query, key, value → multiplies inputs by projection matrices to produce queries, keys, and values matrices
  1. Bottom row of all three matrices is associated with current position, rows above with previous positions
  1. Relevance scoring: Multiplies query vector of current position with keys matrix → score stating how relevant each previous token is
  2. Combining information: multiply value vector associated with each token by that token’s score → output sum of resulting vectors → combining information from all heads
  1. Feedforward layer
  1. Processing power: memorization and interpolation (to generalize beyond inputs not contained in training dataset)

Ch4: Text Classification

Ch5: Text Clustering and Topic Modeling

  1. Convert input documents to embeddings with embedding model
  2. Reduce dimensionality of embeddings with dimensionality reduction model
  1. Methods e.g. principal component analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP)
  1. Find groups of semantically similar documents with cluster model
  1. Centroid-based: e.g. k-means
  1. Requires # of clusters + every data point to be put in cluster
  1. Density-based: e.g. Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN)
  1. Can detect outliers, which will not be assigned to any cluster

Ch6: Prompt Engineering

Ch7: Advanced Text Generation Techniques and Tools

Ch8: Semantic Search and Retrieval-Augmented Generation

  1. Embedding representative (e.g. title, beginning of document) → leaves out lots of information
  2. Embedding document in chunks, embedding those chunks, aggregating chunks into single vector (average of vectors) → highly compressed vector

Ch9: Multimodal LLMs

  1. Both images and text are embedded using image and text encoder, respectively
  2. Calculate cosine similarity between sentence and image embedding
  3. Text and image encoders are updated to match what intended similarity should be (updates embeddings s.t. They are closer in vector space if inputs are similar)
  1. Image-document pairs are used to train Q-Former to represent both images and text (generally captions of images)
  1. Trained on 3 tasks: Image-text contrastive learning, image-text matching, image-grounded text generation
  1. Learned embeddings from Q-former are passed to LLM through projection layer (serve as soft visual prompt)

Ch10: Creating Text Embedding Models

  1. Fine-tune cross-encoder (BERT) using small, annotated dataset (gold dataset): fully annotated with ground truth
  2. Create new sentence pairs
  3. Label new sentence pairs with fine-tuned cross-encoder (silver dataset): fully annotated, generated through predictions of cross-encoder
  4. Train bi-encoder (SBERT) on extended dataset (gold + silver dataset)

Ch11: Fine-Tuning Representation Models for Classification

  1. Sampling training data → generates positive/negative sentence pairs
  2. Fine-tuning embeddings using contrastive learning
  3. Training a classifier (logistic regression by default) using sentence embeddings as input

Ch12: Fine-Tuning Generation Models

  1. Language modeling
  2. SFT → go from base model to instruction (chat) generative model
  1. Full fine-tuning (uses smaller but labeled dataset, as opposed to pretraining) is expensive → PEFT
  2. Parameter-Efficient Fine-Tuning (PEFT)
  1. Adapters: additional modular components inside Transformer that can be fine-tuned to improve performance without having to fine-tune all weights
  1. Adapters that specialize in specific tasks can be swapped into same architecture (if they share same architecture and weights)
  1. Low-Rank Adaptation (LoRA): creates small subset of base model to fine-tune instead of adding layers to model
  1. Decompose large weight matrix into smaller matrices → low-rank version that can be more efficiently fine-tuned
  2. QLoRA: can be improved via quantization: represent original matrix weights by lower precision values → reduced memory requirements
  1. Preference tuning
  1. Collect preference data → train reward model → use to fine-tune LLM
  2. Llama 2 trains two reward models—one scores helpfulness, one scores safety
  3. DPO (2023)—cheaper than RLHF (2017), which requires training RM and LLM
  1. Instead of using RM, let LLM itself do that
  2. Use copy of LLM as reference model to judge shift between reference and trainable model in quality of accepted/rejected generation
  1. Calculated at token level where probabilities are combined to calculate shift
  1. More stable and accurate than PPO during training for human preference tasks (NLP/LLM)

Sinusoidal Positional Encoding

VLLMs

Memory waste in existing systems’ KV caches

  1. Internal fragmentation: over-allocated due to unknown output length
  2. Reservation: not used at the current step, used in future
  3. External fragmentation: due to different sequence lengths

vLLM inspired by OS virtual memory and paging

Paged Attention

Logical and physical KV blocks

Memory efficiency of vLLM

Dynamic block mapping enables sharing

How do PagedAttention and vLLM benefit LLM serving?

Comparisons

Transformer

Attention mechanism

Masked Multi-Head Attention

Next-token probability

Other

Quantization

Activation space

Anthropic’s Persona Selection Model (PSM)

  1. Shoggoth: playacts Assistant persona but only instrumentally for its own reasons
  2. Router: limited non-persona agency in the choice of which persona to enact
  3. OS: predictive model with no agency of its own
  1. Any agentic outputs are due to persona and not underlying LLM

Mixture of Experts (MOE)

“Emotion concepts and their function in an LLM” Anthropic

Dedicated Feature Crosscoder

Tracing the thoughts of a large language model; Anthropic

Autoencoders

Interpretability with Sparse Autoencoders

Model Inference

LLD

LLD = Language-Learning Diary

Features that I want

  1. Login/password reset
  2. Each account has unique “vocab deck”
  3. Game that tests your vocab on how many you can get correct in a row
  1. Give 4 options at a time?
  2. Nlp to get semantic similarity to check if the answer is correct?
  3. “High score” for each account
  1. Diary, can check past diary entries

Preparing Backend 10/22

Goals:

Remarks:

npx tsc

node dist/index.js

“words” table schema

english

diff

freq

pronunciation

spanish

korean

Game Development 11/2

Goals:

Remarks

Authentication + Dictionary 11/12

Goals:

Remarks

Diary 1/9

Goals:

Remarks:

Software Extensibility + Style Polishing 2/8

Goals:

Remarks:

Deployment 2/26

Remarks

Steps to update Docker image

docker build -t yanguages-docker .

docker build → builds new image from instructions in Dockerfile

-t yanguages-docker → assigns name (tag) to image

        . → (build context), current directory

Running Docker image on EC2

sudo docker run -p 8080:8080 -e LLD_PW=[blahblah] yanguages-docker

Steps to update Docker container on EC2

scp -i yanguages-key.pem -r src dist .dockerignore Dockerfile package-lock.json package.json nounlist.csv tsconfig.json ec2-user@54.153.103.184:/home/ec2-user/downloads

FINAL DEPLOYMENT METHOD

RAG project

Setting up RAG

Annoying Bugs

Flow: user enters prompt → ReAct agent reformats prompt to incorporate chat history → calls RAG with reformatted prompt (RAG tool has no chat_history) → output stored in memory ‘output’ key for future chat_history

Multithreading thundering herd problem during scraping

Getting RAG pipeline to correctly output RAGAS metrics instead of nan

Scraping Data

Scraping every harry potter page on hp-lexicon.org with BS4

Scrape every <p> and <li> element within <section> (to exclude navbar)

Retrieve fact_box for each character

Scraping times (without multithreading)

retrieve_magic: 10 min (597 seconds)

retrieve_events: 37 min (2218 seconds )

retrieve_characters: 18.1 min (1088 seconds)

retrieve_places: 12.7 min (761 seconds)

retrieve_novels: 16.2 min (971 seconds)

retrieve_things: 41.43 min (2486 seconds)

retrieve_creatures: 3.28 min (197 seconds)

Total: 154 min (9237 seconds)

Total documents: 5070

Scraping time (with multithreading but no mutex on chroma callback)

Total: 125 min (7504 seconds)

Total documents: 4902

Total documents in Chroma: 4172

Scraping time (with multithreading and chroma callback mutex)

Enhancing RAG

Hybrid retrieval? (semantic similarity + keyword)

Taking Chroma Reranking to the Next Level with a Hybrid Retrieval System | by Sakshi Nepal | Medium

Hybrid Search RAG With Langchain And Pinecone Vector DB

BM25 Retrieval (Best Match 25)

Removed numbering in ReAct framework to not confuse parser e.g. “3. RAG is not a valid tool”

ColBERT (contextualized late interaction over BERT)

Problems

Building Agentic Adaptive RAG with LangGraph for Production | by Piyush Agnihotri | Artificial Intelligence in Plain English

Designing RAG pipelines using LangChain and Evaluating them using Ragas (v0.1.7) | by Rhitesh Kumar Singh | Medium

Retrieval metrics with RAGAS:

https://huggingface.co/datasets/cross-ling-know/HarryPotter-Quiz

Llama-3-8B-Instruct:

Chunking

Creating Dataset

https://arxiv.org/pdf/2405.10166

qa_dataset_llama3_8b

https://openreview.net/pdf?id=pb9qQzAWOS

https://arxiv.org/pdf/2506.04851

Use library json_repair

Random stuff