Information sharing.

• Applying
  • on referrals
  • on resume
  • on research
• Interviews
  • General Programming
  • ML programming
  • System Design
  • Behavioral
  • What i would’ve done differently

I went through a job search loop this July and August, and here is a summary of it for information sharing. This is by no means a guide because there are many things that I don’t know / didn’t do well. I’m sure you can do better than me, good luck!

to start off, the mandatory sankey:

(companies O and A were, alas, in the leftmost grey bar)

Applying

The positions I interviewed for are mostly mid-level and senior MLE (or member of technical staff, if they care to use this name) on the topics of agents / agent infra / continual learning / post training, and a few on forecasting and “world models”, a few on research.

The companies I applied to are around 50% public companies, 30% D+ stage startups, and 20% earlier startups. Among the 22, 3 were recruiter reach-outs, 7 were referrals from friends, and the rest was just applying online from the company’s websites.

on referrals

in my experience this time i find that, unless the referral is strong (e.g. the person has worked with you before and can speak to the process and outcome) and that they know the hiring manager and the exact opening, a referral only gave me slightly better response rate than applying online directly. This might be because I am a more junior and fungible candidate though - my takeaway is ashbyhq works, still apply!

on resume

there is this advice circling on the internet that you have to tailor your resume for each application based on the job description. I didn’t do this, because (1) a lot of the JDs are generic (on purpose), (2) AI can omit seemingly unrelated skills / experiences that would be good to include, (3) AI can hallucinate untruthful details, and fixing them by hand is time consuming. Obviously you should tailor the resume for the job that you want and include the keywords they’d be looking for, but going with one consistent copy of resume did get me decent responses.

on research

my priorities this time were growth and interesting problems. Other than the obvious contenders of frontier labs and star startups, I’ve found opportunities at research fractions of larger startups/companies to be interesting: e.g. perplexity research, decagon labs, scale labs, apple special projects, etc. I enjoyed talking to them - smart people, cool problems, more structure than neolab startups - at the cost of perhaps less security than product-focused roles.

Interviews

roughly 4 types of interviews.

General Programming

• among my non ML programming rounds, only ~30% were leetcode style questions. My hypothesis is that with coding agents companies can make questions that better simulate the real work, which I think is great.
• what I mean by that is more practical coding questions, for example:
  • for an agent role: design + implement an in-memory file system
  • for a search role: implement a simple index + citation mechanism
  • for recsys: debug + edit a code repo for better real time recommendations update
• I only had 1 interview that allowed AI coding tools, and the interviewer mostly observed how I worked with claude (haiku) in a big repo. not sure if that 45 minutes were informative to the company.
  • anecdotally, I know someone who put misleading information in agents.md and tested if a candidate can find out.

ML programming

• Again, very tailored for the role.
• I find these to be hard to prepare for - retrospectively, the best approach would be to know what exactly the team is working on and keep up with recent research on that topic.
• The 1:1 interviews are more or less in this style. Among them, I was asked to implement GRPO once, Gaussian Splatting once (to test GPU parallelism skills, which i did not have enough), and speculative decoding twice (!)
• I preferred the take home assignments more. They are more time consuming, but you get to put in more thought and show more expertise. It also provides perspective on what the team is working on / what they are looking for.

System Design

• Open-ended “build a system that does X” design questions
• Agent / RAG questions are quite straightforward
  • This doc has a good overview of all aspects of a agent system. a bit outdated but it helped me to organize my thinking.
  • specific questions i was asked:
    • chatbot for internal documents (and how to scale to 100k users?)
    • RAG for customer support given policy docs and past interactions (and how to continually improve answers?)
    • memory system for personal agent
    • chatbot for a team working together (multi-user memory / context / interface)
• General ML roles all ask about recommendation systems, with specifics depending on the business they’re in.
  • I am completely new to this interview type. This video helped me with pacing, structure, and level of details in my answers; it also felt more realistic than other resources.

Behavioral

It was interesting to talk to the +1 and +2s and figure out their priorities. Vibes wise: the startups each has a distinct culture, you can feel it just by talking to folks. The big companies felt more chaotic and overwhelmed, at least compared to a couple years ago. The two questions I usually go for are (1) what are the most challenging / annoying things about working here, and (2) the vision of where the product / research / team is going in near and far future.

I might be an idealist, but I get the ick when a team’s priority is to defend scope and collect paychecks for as long as possible; I want to find a place where we build.

What i would’ve done differently

these are mostly for myself and are subject to further reflection.

• reach out to more people. The two positions I interviewed at apple were not the best fits for my skillset - so i didn’t get them - but by telling the stories it turns out that I knew quite some people working on ML at apple, and they pointed me to better fitting teams and projects, plus companies where good managers have gone to start teams. By the time this happened I was already in the offer negotiation stage, so here’s a reminder to talk to more people earlier on in the process.
• Do more thorough prepping for ML coding, for each of the positions. The ML rounds were more in-depth and up-to-date than I expected, and if I had the time / mental capacity I would dig deeper into the field I am going into, read up, implement the common algorithms, and know the trade-offs.
• work on perspective and signals for corporate america: leadership, ownership, conflict and stress handling, etc.

动·玩·生 (Learning to Move, Learning to Play, Learning to Animate) at Inclusion Conference on the Bund in Shanghai.

Our work is a part of the art exhibition at the Inclusion Conference on the Bund in Shanghai this year. Mingyong, Han, Lindsey and I presented together as the artist group 0x0a. Huge thanks to the curation team ;C for their support.

Some stills from the event:

For more about the project, see the project blog post.

In support of nice digital artifacts as tokens of appreciation.

I submitted a review for a 70 page JMLR manuscript the other day, and they brought me to a page that expressed gratitude and gave me a zip file to download. I openned the zip and found a README.md file. The readme went:

# badges
I review for JMLR badges

Here’s the badge in question.

cute and/or cringe, seeing it sparked some 2000s internet joy in me.

It also reminded me of Alex Danco’s piece on how science, open sourced software, and startups benifit a lot from gift culture. The reciprocity and goodwill of gift exchange culture (rather than, say, the market of labor and capital) encourages risk-taking and genuine contribution.

Is this badge a gift? It signals that I’m participating in the community around scientific reciprocity, which i proudly displayed in this blog post for vanity. It does not compensate for the 6 hours I spent on it, but is nontheless fulfilling. Maybe one small reason for ML conferences’ disfunctional review system is the lacking of social credits given to reviewers? (I know review experience helps with visa & immigration, but that would not encourage service beyond the needed amount.) Can we start giving out little internet trinkets, unique each year, like ELO badges from games?

I look forward to seeing gifs for reviewers from more ML conferences.

An immersive performance featuring multi-modal generative machine learning model @ UCSD’s Qualcomm Institute.

This is a project funded by UCSD’s Qualcomm institude IDEAS program and is a collaboration with many folks here. See full list of contributors below.

Trailer

What does it mean to be intelligent? Is it unique to humans, or can it take other forms—beings of wood, stone, metal, and silicon? Recently, we have seen rapid advances in “artificial intelligence,” but its definition has remained elusive. AI systems are often portrayed as some “other,” an alien invention that threatens to overpower us.

At the same time, we are only just becoming aware of the other intelligences surrounding us on this earth, those we’ve failed to recognize or acknowledge for so long. These beings—animals, plants, and natural systems that have been with us all along—are slowly revealing their complexity, agency, and knowledge.

Ecologist and philosopher David Abram coined the phrase “the more-than-human world” to refer to a way of thinking that seeks to override our human tendency to separate ourselves from the natural world. Our work is the artists’ reflection on learning to see technology and nature as part of this more-than-human world. What does it mean to live in a world where we, the technologies we have created, and nature form a collective intelligence? How do we coexist without overpowering or being overpowered by one another?

We explore these concepts in our performance through mediums and cutting-edge technologies including self-developed organic material robotics, AI-generated visuals, audio spatialization, real-time bio-feedback data sonification and visualization, and movement. Our performers, machines, and visuals interact in the realm of shadows, whose conceptual framework be traced back to Plato’s allegory of the cave, where shadows on the wall represent the limited and often deceptive nature of human perception. Beginning with mutual imitation and later progressing to playfulness, our performance reflects the complex process of how humans, nature, and machines perceive and learn from one another, reflecting the connected consciousness of our environment, our technologies, and us.

Organic form Robots

Large center robot, Jensen’s linkage.

Arduino IoT controlled organic robot arm, and the shadow it casts.

Full docummentation:

Credits

(in alphabetical order):

Co-Directors: Mingyong Cheng, Sophia Sun, Han Zhang

Performers: Yuemeng Gu, Erika Roos

Robotic Engineer: Sophia Sun

Visual Artist: Mingyong Cheng

Sound Designer: Han Zhang

Lighting Engineer: Zehao Wang, Han Zhang

Video Editor: Yuemeng Gu

Post Production Coordinator: Mingyong Cheng

Technical & Installation Support: Yifan Guo, Ke Li, Zehao Wang, Zetao Yu

Special thanks to Palka Puri for plant support, the Initiative for Digital Exploration of Arts and Sciences (IDEAS) program at the University of California San Diego and Qualcomm Institute for sponsoring this project, and the AV team from the California Institute for Telecommunications and Information Technology (Calit2) for installation and media support.

Poster at Neurips 2024.

a.k.a. Richard Hamming vs. Richard Feynman

–

I am going to graduate my phd soon. I’ve been talking to some prospective students recently, and a common question that comes up is: what makes a good phd student? how to be a good researcher?

I’d point them to some articles, but to be honest I have no idea. What I did spend a lot of time thinking about, and have arrived at a somewhat self-convincing answer, is to a slightly different question - what makes a happy phd student?

The answer is that they are married.

(Obviously this is a joke. We did once survey a 10-phd bbq party, and marriage was the most statistically significant positive correlator. But that’s a story for another time.)

Richard Hamming’s inspiring and influential talk You and Your Research is what convinced me to do a phd. I wanted to work on important problems and aspired to contribute to the advancement of human knowledge. Oh, to be a scientist! 3 years in I felt like there was no hope; I’ve toiled away all my time and the only way out is to quit.

So I sought advice from my parents. They asked me to think about the least accomplished PhD graduate I know - Can I do as much as they did? What would a minimum effort phd look like? I answered that would result in a pretty bad thesis. And they said, that’s fine, bad is ok, go do what you can. Anything worth doing is worth doing badly.

And here we are.

This line of reason was put into words a lot more elegantly by Richard Feynman. Allow me to quote his letter in its entirety here - I thought it was the perfect antidote to Hamming-esque anxieties, the Dionysus to our Apollo. Enjoy, and hope you will finish your phd with peacefulness in your heart as well.

–

Dear Koichi,

I was very happy to hear from you, and that you have such a position in the Research Laboratories. Unfortunately your letter made me unhappy for you seem to be truly sad. It seems that the influence of your teacher has been to give you a false idea of what are worthwhile problems. The worthwhile problems are the ones you can really solve or help solve, the ones you can really contribute something to. A problem is grand in science if it lies before us unsolved and we see some way for us to make some headway into it. I would advise you to take even simpler, or as you say, humbler, problems until you find some you can really solve easily, no matter how trivial. You will get the pleasure of success, and of helping your fellow man, even if it is only to answer a question in the mind of a colleague less able than you. You must not take away from yourself these pleasures because you have some erroneous idea of what is worthwhile.

You met me at the peak of my career when I seemed to you to be concerned with problems close to the gods. But at the same time I had another Ph.D. Student (Albert Hibbs) was on how it is that the winds build up waves blowing over water in the sea. I accepted him as a student because he came to me with the problem he wanted to solve. With you I made a mistake, I gave you the problem instead of letting you find your own; and left you with a wrong idea of what is interesting or pleasant or important to work on (namely those problems you see you may do something about). I am sorry, excuse me. I hope by this letter to correct it a little.

I have worked on innumerable problems that you would call humble, but which I enjoyed and felt very good about because I sometimes could partially succeed. For example, experiments on the coefficient of friction on highly polished surfaces, to try to learn something about how friction worked (failure). Or, how elastic properties of crystals depends on the forces between the atoms in them, or how to make electroplated metal stick to plastic objects (like radio knobs). Or, how neutrons diffuse out of Uranium. Or, the reflection of electromagnetic waves from films coating glass. The development of shock waves in explosions. The design of a neutron counter. Why some elements capture electrons from the L-orbits, but not the K-orbits. General theory of how to fold paper to make a certain type of child’s toy (called flexagons). The energy levels in the light nuclei. The theory of turbulence (I have spent several years on it without success). Plus all the “grander” problems of quantum theory.

No problem is too small or too trivial if we can really do something about it.

You say you are a nameless man. You are not to your wife and to your child. You will not long remain so to your immediate colleagues if you can answer their simple questions when they come into your office. You are not nameless to me. Do not remain nameless to yourself – it is too sad a way to be. now your place in the world and evaluate yourself fairly, not in terms of your naïve ideals of your own youth, nor in terms of what you erroneously imagine your teacher’s ideals are.

Best of luck and happiness.

Sincerely, Richard P. Feynman.

Point predictions are often insufficient for machine learning tasks that involves uncertainty. As a solution, probabilistic models are becoming more widely used. This post talks about proper scoring rules, a framework to think about and evaluate probabilistic predictions.

• Probabilistic Predictions / Predictive Distributions
  • Calibration and Accuracy
• Proper Scoring Rules (PSR)
  • Definition
  • Common Scoring Rules
    • Negative Log Likelihood (NLL)
    • Brier/Quadratic Score
    • Continuous Ranked Probability Score (CRPS)
    • Energy Score
    • Interval specific: Mean Interval Score
    • Interval specific: Winkler Score
    • tangent: Is KL divergence a proper scoring rule?
• Discussion

Probabilistic Predictions / Predictive Distributions

Predictive distributions are when a model produces a probability distribution (as opposed to a point prediction) given data and parameters. They have many benefits - many real world problems have inherent uncertainty, and we can capture thing we care about, like risk or reward, through a probabilistic representation. With predictive distributions, we can have better uncertainty quantification, and make better decisions given these uncertainties.

To put in perspective, for input \(X\), instead of outputting a point prediction \(y\), the model outputs a distribution over the space of outputs \(p(y\vert X)\).

For example, for a classification task, we can output confidence along with a label:

For regression, one can output a gaussian distribution over \(y\) (mean and variance), like in the case of Gaussian Process regression:

Uncertainties can be from different sources. The most mainstream distinction is aleatoric (there exists variations in the data, as in our classification example) vs epistemic (we don’t have enough data to know the label for sure, as in our regression example).

Calibration and Accuracy

In general, we want two thing from our probabilistic forecasts: calibration, and accuracy. Calibration refers to that the probabilities predicted actually corresponds to the true probabilities. For example, among the days that we predict 30% precipitation rate, 30% of them actually rained.

In the machine learning community, many studies have shown that as neural networks get deeper, they become less calibrated. So there have been efforts to calibrating neural networks using techniques like ensembles and temporal scaling, among others. This is still an active area of study.

An important note is that calibration is not the same thing as accuracy. A model can be perfectly calibrated but completely useless - for example, for a binary classification task with a balanced test set, a model that predicts class A always with \(50\%\) probability.

Evaluating probabilistic forecasts, then, requires evaluating both accuracy and calibration. This is where proper scoring rules come in.

Proper Scoring Rules (PSR)

Definition

Scoring rules are functions that score how well a predictive distribution captures data. Let’s say we have data domain \(x \sim \mathbf{X}\), and \(P\) is a distribution over \(\mathbf{X}\). Then the score

It assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution \(F\) if he or she issues the probabilistic forecast \(F\), rather than \(G = F\). It is strictly proper if the maximum is unique.

Common Scoring Rules

Negative Log Likelihood (NLL)

You probably know what NLL is. In addition to being a proper scoring rule, it also have optimal discrimination power.

Brier/Quadratic Score

Brier score is a PSR for classification tasks. It is sometimes preferred over NLL because it is bounded within \([0,1]\) and hence more stable.

In the binary setting, we can calculate it as:

• \(n\) is the total number of observations or data points. ​

• \(p_i\) represents the predicted probability of the positive class for the i-th observation.

• \(o_i\) is an indicator variable that equals 1 if the event occurred and 0 if it did not for the i-th observation.

Continuous Ranked Probability Score (CRPS)

Continuous Ranked Probability Score (CRPS) is another metric widely used in fields like meteorology, hydrology, and climate science. In plain words, the CRPS measures the discrepancy between the predicted cumulative distribution function (CDF) of a forecast and a step function representing the true outcome. It essentially quantifies the spread of the forecast distribution around the observed value.

The Continuous Ranked Probability Score is defined as:

• \(CRPS(F, y)\): The Continuous Ranked Probability Score for the forecast \(F\) and the observed outcome \(y\).
• \(F(x)\): The cumulative distribution function (CDF) of the forecast, evaluated at \(x\).
• \(\mathbb{1}(x \geq y)\): The indicator function which equals 1 if \(x \geq y\) and 0 otherwise.
• \(dx\): Infinitesimal change in \(x\), indicating integration over the entire real number line.

Energy Score

Energy Score (ES) is a proper scoring rule to measure calibration and sharpness of the predicted distributions. Defined as

This paper shows that the energy score outperforms the softmax confidence score on common OOD evaluation benchmarks.

The energy score is parameter-free measure, which makes it easy to implement, especially for distributions whose analytic expressions are unavailable or difficult. Another result of this property is that it is not directly differentiable w.r.t. parameters.

Interval specific: Mean Interval Score

A common way of using probability forecasts for decision making is translating it to confidence intervals. We will go through two of the

The Mean Interval Score is a metric used to evaluate the accuracy and calibration of probabilistic forecasts. It measures the average width of prediction intervals relative to the observed outcomes.

Keep in mind that this is a specialized metric for evaluating prediction intervals, and it may not be as commonly used as metrics like Brier score or log-likelihood in all contexts.

In a prediction interval, a model not only predicts a point estimate but also provides a range of values within which the actual outcome is expected to fall with a certain confidence level.

The Mean Interval Score is defined as follows:

Where:

• \(n\) is the total number of observations or data points.
• \(U_i\) is the upper bound of the prediction interval for the i-th observation.
• \(L_i\) is the lower bound of the prediction interval for the i -th observation.
• \(y_i\) is the actual observed value for the \(i\)-th observation.
• \(\alpha\) is the confidence level of the prediction interval (e.g., 0.95 for a 95% confidence interval).
• \(I(\cdot)\) is the indicator function, which equals 1 if the condition inside the parentheses is true, and 0 otherwise.

The Mean Interval Score penalizes prediction intervals that are too wide or too narrow compared to the actual outcomes. It also accounts for cases where the actual outcome falls outside the predicted interval.

Interval specific: Winkler Score

The Winkler score is another fun one for confidence intervals / regions, and can be easily extended to forecasts in higher dimensions. It is essentially a a metric measuring the overlap between two sets.

where \(A\) and \(B\) are binary sets and | \(\cdot\) | denotes the cardinalities of the sets.

The Winkler Score ranges from 0 to 1, with 1 indicating complete overlap and 0 indicating no overlap between the sets.

tangent: Is KL divergence a proper scoring rule?

No. The main difference between KL divergence (and other entropy metrics) as opposed to PSRs is that it measures difference between two distributions, where as PSRs calculates a score with the predictive distribution and observations. Often we do not have the ground truth distribution, so we would have to approximate the KL divergence.

KL is also not strictly proper: While KL divergence is proper in the sense that it is minimized when the predicted distribution matches the true distribution, it is not strictly proper because it doesn’t have a unique maximum. In other words, multiple predicted distributions can yield the same KL divergence value.

This is not to say that we should not use KL divergence as a measure of accuracy. It is very helpful for optimization and model selection because of its sensitivity to small differences, and can be used along with other PSRs.

Discussion

I hope I’ve made the case for using proper scoring rules as evaluators of probabilistic predictions, and provided a grimoire of useful scores.

It is worth noting that, while being proper is helpful for model comparison, it is just one of the desirable property for a metric. This paper from ICML 2023 raised the important point that we also need to pay attention to metric’s discriminative performance in the settings of the task. They showed that in a multivariate time series forecasting context, the energy score and CRPS fail to capture correlation structures between the variables, and are not sensitive to discrepancies in higher moments. Some scoring rules also appears complementary in their discriminative emphases, so it is worth having an ensemble of metrics for reliably measuring the correctness of forecasts.

Happy hacking!

Extended Readings

• Strictly proper scoring rules, prediction, and estimation.. Gneiting, Tilmann, and Adrian E. Raftery. The OG proper scoring rules paper.

• Valid prediction intervals for regression problems. Nicolas Dewolf, Bernard De Baets, Willem Waegeman. A survey of probablistic forecasting models including Bayesian methods (GP, BNN), ensemble methods, direct estimation methods (quantile regression), and conformal prediction methods. They introduce the methods clearly and compare the advantages and disadvantages of each.

References

Michel Kana, “Uncertainty in Deep Learning. How To Measure?” blogpost, 2020

Balaji Lakshminarayanan, Dustin Tran, and Jasper Snoek, “Uncertainty and Out-of-Distribution Robustness in Deep Learning” Talk at NERSC youtube, 2020

Gneiting, Tilmann, and Adrian E. Raftery. “Strictly proper scoring rules, prediction, and estimation.” pdf Journal of the American statistical Association 102.477 (2007): 359-378.

Victor Richmond Jose, Robert Nau, & Robert Winkler. “Scoring Rules, Generalized Entropy, and Utility Maximization” slides, Presentation for GRID/ENSAM Seminar, 2007

Marcotte, Étienne, Valentina Zantedeschi, Alexandre Drouin, and Nicolas Chapados. “Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts.” pdf ICML 2023

Update: We presented ur music video for Lockdown at the Machine Learning for Creativity and Design Workshop at NeurIPS 2022.

our poster for NIME 2021  

You can read the paper here and listen to our music here.

This paper documents the music-making of the band Seals, for which I play bass and do machine learning. The Seals is a pet project with Margaret Schedel, Susie Green, Ria Rajan, and Sofy Yuditskaya; we introduce ourselves, (cheekly), as a political, feminist, noise, and AI-inspired electronic sorta-surf rock band. The S.E.A.L. (Synthetic Erudition Assist Lattice) is the collection of algorithms that assist us in creating content with which to mold and shape our music and visuals. Glad to see our acronym powers from academia came into use.

Our first collection of songs are about how our living experiences are mediated by technology, pandemic lockdowns, the female body, and the occult. I hope we can bring a performance to you one day.

Sofy made custom theremins for all of us.  

I built a drawing robot with wood found in the desert! The machanism and software is heavily based on BranchioGraph, with a different construct and orientation that required some tinkering to figure out.

very ad-hoc calibration setup ft. printed protractor and earing

first run! It drew a bonfire.

Even after calibration the drawing is still a little… wonky, because of the elasticity in the sticks, the looseness of the joints, and inacuracies of servo controls. In some ways, though, these faults added a layer of intermediacy to the drawing (or the act of drawing), which one may see as part of the machine’s touch.

A drawing of a vectorized picture of us building pallet racks.

The bot even participated in our weekly drink-and-draw event. People blew into a alcohol sensor appended to the raspberry Pi, whose reading were used to parametrize the open-cv vectorize function. (So technically, we drank and it drew.) Here is a video of the bot figure drawing.

  You may find my code for it here.

Pieces of writings that I found helpful for thinking about research and surviving a PhD.

De-Mystifying Good Research and Good Papers, by Fei-Fei Li

read it here.

I’m putting this one first because it’s a succint yet powerful read. It’s easy to be caught up in the “publish or perish” culture and the race of metric hacking / low-hanging fruits; this article serves as a good reminder of that’s not why we are here doing research.

Every research project and every paper should be conducted and written with one singular purpose: to genuinely advance the field of computer vision. So when you conceptualize and carry out your work, you need to be constantly asking yourself this question in the most critical way you could – “Would my work define or reshape xxx (problem, field, technique) in the future?”

This means publishing papers is NOT about “this has not been published or written before, let me do it”, nor is it about “let me find an arcane little problem that can get me an easy poster”. It’s about “if I do this, I could offer a better solution to this important problem,” or “if I do this, I could add a genuinely new and important piece of knowledge to the field.”

You and Your Research, by Richard Hamming

read it here.

A classic and important reading for any researcher in science. In his talk (linked is a transcription) Hamming talks about what constitutes important work, why do it, and how to do it. The arguments are interlaced with cool personal science anecdotes and lined with optimism – it reminded me of why I am here doing grad school in the first place. I first heard about the talk’s existence, funnily enough, from Paul Graham, but I’ve seen scientists I look up to reference it again and agian, from when they’re feeling burnt out to when facing important decisions. It carries a message worth bearing in our minds as we go about, well, being us and doing our research.

The PhD Grind, by Phillip Guo

(update: Phillip took down the book; see his FAQ. you may still find copies online. Perhaps instead, check out his Advice for early-stage Ph.D. students.)

The pitch I generally give people about this book is that Phillip encountered a pretty comprehensive set of problems for a CS PhD student, and prevailed (!). In this mini memoir he writes with clarity, provides sound analysis, and offers precious advices. Whatever problem you are experiencing, you are likely to find a way to reason about it from this book. Feel like you are just doing grunt work? Find it hard to communicate with you advisor? Anxious about not publishing (whereby proving your competence)? Deciding between industry vs academia? Phil’s got you.

It is both endearing and inspiring to learn about his journey, and to know that he came out ok (he’s a professor at UCSD now!). I really appreciate the honesty and detail he put into this work. Thank you!

Other Interesting things

• An illustrated guide to PhD
• How to transform into a senior PhD student
• A PhD Is Not Enough!: A Guide to Survival in Science (book)
• phd comics

This is a computer-generated karaoke I built with Sofy and Derek during our time together in Brahman.ai, Bombay Beach CA.

For details, please see our paper :)

  Sofy building the tesseract.