Mulan’s talk with her dad about moving out
Mulan had a long discussion with her dad, where she told him his plan.
- Her dad first said it’s dangerous and stuff
- Then asked her what if he came with her
- Discussion came to the possibility of him leaving her, when kids are gone
- Wouldn’t want to completely dip, but don’t want to have contact with her for a few years
- Is she asking him that she’s going to go? Or just telling him that he’s going to go?
- Basically decided, wasn’t even going to tell him, but telling him because of respect for him, because he’s a pretty good parent
- Discussion of Jun as a parent
- Good parent, pretty strict
- Mulan wished he would’ve fought more
- He says he fought in many ways, but didn’t really work
- They both realized it was a solutionless problem
- Earlier, he was thinking what if he left, and took the kids
- Also thought about leaving, but didn’t trust her with the divorce, didn’t think she was capable of taking care of the kids
- Later got complacent with it, because didn’t think that there was anything else that could’ve been done
- Everything that he was doing was for the kids
- Counterarguments to Mulan going 202006051006
- Protests
- Schools might open in 3 months
- Maybe it’s because you’re stuck at home too much
- Asked around, and people are willing to let her stay at their places
- COVID is a burden, wouldn’t want to add a burden on people
- You can stay here, and you don’t have to talk to her, you don’t have to apologize to her
- It’ll be like now, you just focus on your work
- Why do that, when I can do something better?
- Might be a big mistake
- Why do that, when I can do something better?
- You can just move out after a year
- It’ll be like now, you just focus on your work
- You wanted to go to Berkeley anyways, you just used this huge fight as a tipping point
- Summary: A lot of stuff going on, food, etc, don’t have to cook, just focus on work
- Would rather have the stress of fending for myself, than dealing with this situation
- What are the nuances of the situation
- Would rather have the stress of fending for myself, than dealing with this situation
- Things are different with Lilian, because she’s like that all the time.
- Things are different with Jun, because he forgets pretty fast, and there’s a lot of stuff with the house that gets paused when you’re not talking with each other, so it can’t last very long
- Also, he tends towards inaction generally
- Pretty hard to fix
- If you wanted to, you could try to fix it
- At the same time, you can’t start doing it instantly
- He understands that he has the image of being weak/passive
- Actually gets angry pretty fast, but pretty good at holding things in, and tolerating them
- He thought that it makes him weak
- Mulan says that makes him strong brave for other people
- Also, he tends towards inaction generally
- Mom is someone who can only think in terms of emotional/personal stuff.
- Realizing that as she gets older, she’s similar to her mom
- Dad said yeah, but you’re way better at self-reflection, which is what I say all the time too
- Mentioned a couple of things she did when they were younger, and it made Jun really sad
- If something bad happens, Jun just brushes it off, but it made him realize that these wounds don’t just go away
- When Jun and Liz first met, she was fighting with grandma, uncle, etc.
- Mom’s family would cut off contact for years, and some day something would happen, and they would bring it back super fast
- When mom went with cousins to Napa, huge fights
- There is no one that she isn’t a bitch with, including her own family
- When mom gets in fights some other people, and Jun says something in support of other person, immediately becomes about how he’s “betraying her”
- Even in disagreements with grandma, if he’s supporting grandma, also betraying
- When he used the wrong dish sponge, she just threw the sponge in the sink and started yelling at him
- Mulan realized that if dad wasn’t angry, then she won’t cause a scene
- He was actually really angry, just didn’t think was worth it
- Interesting to think about what happened with her
- Was like that when they met, which shows that Jun has issues too
- Jun’s side of the family was very traditional and respectful
- Thought she might mesh in that kind of environment
- Can’t just change someone
- Jun told Mulan to not lose connection with people in Dongbei
- Find Helena, the Australian cousin
- Don’t let issues with the generation before you let you lose contact with people from your current generation
- When Jun was younger, he would get really drunk, and fought with grandmother
- Would make him very nervous, and he would hide, so he understands what’s going on with Mulan
- Had discussion about politics again 202006080251
- This time, the tone was very different 202006020038
- Doesn’t think a lot of these issues will work, because left is so extreme
- People are afraid of admitting they’re not PC, because they’ll get attacked
- A lot of conservatives get attacked
- Separate thing: People at Facebook are so transparent 202006042307
- People were posting publicly, while Zuck was giving his speech, about how they’re so disappointed in him
- Dad asked if Mulan agrees with Zuck
- Zuck’s post 202006042310
- Poor mans digging himself into a hole
- Doesn’t really agree with him
- If you just limit to the statement about the police system being racist, then he would agree with that, it’s true, the statistics are true
- I just don’t want you to be simple-minded, and I don’t want you to have a singular viewpoint
- Mulan never things that one side is ever completely correct or incorrect
- But when you were arguing against it, Mulan got mad because they were being extreme, and she was extreme in response
- True of arguments in general: 202006042313
- But when you were arguing against it, Mulan got mad because they were being extreme, and she was extreme in response
- Democrats think you have a responsibility to help people, vs Republicans think that people can fend for themselves 202006042317
- Jun thinks that affirmative action helps one group of people, while screwing over others (Asians) 202006050127
- Did admit that the black people who made it out of poverty usually end up really good people
- Have go to and encourage the people to have jobs
- Spend the money on education, make sure teachers are paid more
- People are against the Democrats because they funnel everything into social welfare, instead of institutions
- Institutions should work, but a lot of the time it doesn’t work
- On the other hand, just socially helping people out doesn’t always work either 202006042317
- See 202006042317 for other thoughts
- Jun thinks that the debate about transgender bathrooms is dumb, because it’s not that bad for people to have to go to the women’s bathroom
- You should be able to handle it first of all, and bathrooms are built a certain way, because of convenience. It’s not that big of a deal
- Single gender bathroom is fine, he agrees with it too
- Doesn’t understand why it became a policy issue, seems extraneous
- Transgender community was pushing for small things first
- Going back to the original point: Jun says that he can’t stop Mulan from doing it, but think about it a little more, and maybe wait out the protests
- Mulan says that she still really wants to do it
- A bunch of other people are going back to Berkeley, just because they’re bored
- That’s incomprehensible to Mulan
- Mom still treats them as children
- That’s incomprehensible to Mulan
- Not even like she’s running away, pretty normal
- Jun says it’s different because you’re basically running way, and it’s going to have consequences
- It’s not an easy thing, and you’re still going to feel guilty, and it might make you feel worse
- Mulan doesn’t seem like there’s any point in waiting, it’s always just going to end up happening
- Jun admits that there’s no solution, he thinks Mulan should go, but doesn’t know how it could happen
- Mulan still really wants to go
- The longer she waits, the longer thinks might go back to normal
- Mulan is scared that if the situation goes back to normal, then she might lose her motivation
- Understands that it’s kind of like cheating her way out, but she’s still kind of nervous about that
- Mulan was really grateful that he let her just talk to her
- Lilian won’t care, she won’t be torturing herself over her for very long
- Lilian doesn’t bottle things in, she’ll probably just fight her
- Mulan told Jun to try to talk to Lilian more, so she feels comfortable having these kinds of conversations with him
- Jun doesn’t think that disappearing is even that crazy, even for Lilian
- Now is just not the best atmosphere for it
uid: 202006042233 tags: #talks
Walk with Yash 5-31-20
Walked with Yash, and had a lot of great conversations and insights, as always. Here’s a small collection of topics that we discussed, based on what I can remember the day after:
- I talked to him about the Algorithms book I’ve been reading (202005302204), and practiced conveying my ideas about something that I’ve read and supposedly digested to other people. I think it went pretty well.
- We talked about our time with comp civics (and also Peter Ernst)
- We talked about mortality, and the WeCroak app
- We talked about weed, and how Yash has a bunch of thoughts going through his mind, and he’s pretty good at getting them out when he’s talking about them, but they don’t really come out otherwise.
- We talked about tech companies, how dominant they are, and why Microsoft isn’t at the same pedestal as Google, Facebook, Apple, Amazon (it’s because their peak was when tech was more of a niche than it is today.)
- We talked about how Uber and Lyft grew to occupy the same niche (when Lyft first started out, they had to figure out how to be slightly cheaper than Uber, but also have sufficiently high quality that people wouldn’t just never take their product. It was also the fact that people were a little more trusting than they are today, so they were willing to try a totally new product that didn’t have a wealth of reviews already built up for drivers on their platform).
- We talked about how medicine is moving towards palliative and hospice care a lot recently, over the last 20 years. There is more and more of an emphasis on doctors not just being these mechanical people, who just have tools to fix a person’s body, but also improving their mental health and overall wellbeing as well.
- Met Holly Cheek along the way, which was slightly awkward. Interestingly enough, she transferred to Berkeley from UCSB, which I didn’t realize was a thing
- Talked about of course everything that’s going on 202006031231, and the combination of the fact that things are just getting to a natural tipping point, and a lot of people are at home/unemployed so they don’t have too much else to do, that things are just exploding all across the country.
- Talked about being happy with what you have (I told him that I got amazon and DS100)
- Talked about how we’re both fundamentally good people. Yash mentioned that there are like 5 people (and the rest are much older than me) that he can have deep conversations with like the ones that he has with me, and I show a considerable amount of mental and emotional maturity. I think it might be one of my greatest strengths - this calmness and maturity.
- Talked about TV shows briefly towards the end — Yash like Seinfeld a lot, and is currently going through Arrested Development
- Yash accidentally dipped his foot in some mud, and so for the last few minutes he was walking while holding his chappal in his hands, true Indian style. I mentioned, what would happen if someone just dressed as an Indian beggar and walked across Pleasanton for an entire day. I wonder how people would react, and if people would be racist/disgusting.
- The fact that I’m just learning about the different streets in the neighborhood, and how going for runs is great for that.
- In the beginning, we talked about the 10k that Yash did, and how that in the beginning, why he would struggle with running was because of his heart and lungs, but at this point, it’s the fact that his legs are literally so tired that it’s hard to move them.
- Yash has struggled with anxiety for a while - basically because he tends to take on too much for himself, and tries to do too much to help people. Over the last couple years, he’s been trying to bring things back into his “locus of control” - only focus on the things in his immediate vicinity, like his family and friends, and try to help them as much as possible, while not putting on the burden of the world onto himself.
- Somewhat related, but he talked about how in his childhood, he was pretty naive about America - he thought that like him, people were mostly out to help other people, and generally wanted good for each other. Now that he’s realized that that’s not really the case, it’s allowed him to be much more at peace with his view of himself and America.
- Yash doesn’t take on too much, but when he does, he has really high standards for himself. This came up in the context of him talking about how he spent a long time on all his English essays, even all throughout college. He said that he wouldn’t really write for himself though.
Also talked about the disconnect
uid: 202006010927 tags: #insights #talks
Manan Khattar
Internship Retrospective
Friday, May 29, 2020
Some Stuff I Worked On
- Built in Markdown functionality in Datanav #3734
- Put the final nail in Airflow’s coffin #4048
- Created a
KafkaMockmodule for convenient testing. #4205 - Ported the Decline ingester to the loader framework #4505
- Created a Hypothesis testing framework to test Redshift loader processing scripts #4769
Datanav Demo
Turning Off Airflow
- Converted existing
airflow_dagsin BUILD files toargo_dags - Some were easier than others - most complicated ones were the FTP and Decline ingesters
- Made final nuking PR - removed 99% of all instances of Airflow from the codebase
- Worked with Josh K. to delete Airflow Kubernetes deployment/namespace, delete the Airflow database from Postgres, etc
Turning Off Airflow
Only 16 instances of Airflow in the codebase! (Mostly old documentation/historical artifacts).
right
Kafka Mock
Setup
- Writing tests for ETLs is hard, especially if they interact heavily with Kafka
- Confluent Kafka is annoying to use
- Josh K. made a wrapper for the Kafka functionality we actually use (
py6/util/kafka.py) KafkaMockshould have the same functionality as theKafkamodule (but work in local memory)
[.build-lists: false]
Kafka Mock
YAKs
- Wrote a script to download sample (non-test-id) messages from a given topic (
py6/kafka/inspection.py)
def download_messages(topic_name,
group_id="message_downloader",
cluster_name="staging",
num_messages=20,
filter_func=None):
# type: (str, str, str, int, Optional[Callable[[kafka.Message], bool]]) -> None
"""
This script can be used when one wants to download sample messages from a specific Kafka topic,
for use in testing and scripts that test ETLs.
The filter function cannot be passed in through the command-line, so the command-line interface
to this function has no filtering capability by default. When called programmatically, however,
the filter function can be used to filter out the Kafka messages that shouldn't be downloaded
from the topic.
"""[.build-lists: false]
Kafka Mock
Intermediary steps
- Incrementally passed the unit and integration tests of the
Kafkamodule
alpha/py6/testing on manan/hypothesis-testing-for-loader [$?] via thm3 via 🐍 system
➜ pytest /Users/manankhattar/alpha/py6/kafka/test_kafka_mock.py
=================================================================================================================================== test session starts ====================================================================================================================================
platform darwin -- Python 3.7.6, pytest-3.5.0, py-1.8.1, pluggy-0.6.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/manankhattar/alpha/py6/testing/.hypothesis/examples')
rootdir: /Users/manankhattar/alpha, inifile: pytest.ini
plugins: hypothesis-4.32.3, Flask-Dance-2.2.0, flaky-3.4.0
collected 22 items
../kafka/test_kafka_mock.py ...................... [100%]
===Flaky Test Report===
===End Flaky Test Report===
================================================================================================================================ 22 passed in 0.11 seconds =================================================================================================================================Kafka Mock
[.build-lists: false]
- Mostly works out of the box! (A bit of setup/monkeypatching required)
[.column]
@pytest.fixture(scope="module")
def setup(cluster_name):
# type: (str) -> None
# creates a mock cluster and a mock topic
curr_cluster = kafka_mock.new_cluster(cluster_name)
curr_topic = kafka_topics.ProsperDeclineCallsTopic
json_file_path = os.path.join(
os.path.dirname(__file__),
"sample_messages_decline_program.prosper.ProsperDeclineAPICall.api_calls.json")
# add topic to cluster
curr_cluster.ensure_topic(curr_topic)
# adds messages from json to the new topic
kafka_message_list = json_to_list_of_messages(json_file_path, curr_topic)
# makes a new mock producer
curr_producer = kafka_mock.Producer( # type: ignore[type-var]
topic=curr_topic,
cluster="staging",
)
# mock producer sends messages from json to mock cluster
for message in kafka_message_list:
curr_producer.reliable_send(message.value(), message.timestamp())[.column]
@pytest.fixture(scope="function")
def mocking_and_monkeypatching(monkeypatch):
# type: (_pytest.monkeypatch) -> None
# do monkeypatching
monkeypatch.setattr(kafka, "Consumer", kafka_mock.Consumer)
monkeypatch.setattr(kafka, "Authenticator", MagicMock())Decline Ingester / Loader framework
Setup
- Previously, the decline ingester had its own processing script (
py6/etl/scripts/ingest_decline_calls.csv) - Written before we decided on a common framework for all scripts that load messages from Kafka and put them in Redshift (
kafka_to_redshift.Loader) - Take logic from
ingest_decline_callsand put it into a single processing function:(kafka_to_redshift.ProcessorDataType) -> pd.DataFrame
Decline Ingester / Loader framework
YAKs
- Usage of
argparsein thekafka_to_redshift.Loadermade it difficult to write tests (couldn’t directly pass in command line arguments) - Some discrepancies in the arguments that could be passed into the loader programmatically vs from the command line
Decline Ingester / Loader framework
YAKs (Small changes to kafka_to_redshift.Loader)
- Extracted all argument parsing functionality from the
runmethod inkafka_to_redshift.Loader - Create a class method,
kafka_to_redshift.Loader.parse_args, that can parse command line arguments and then pass them to the initializer - Make all arguments in the
__init__method of the Loader required. (Used for testing)
Decline Ingester / Loader framework
/ End YAK digression
[.column]
- Made 4 new loaders - 2 each for LC and Prosper, prod and staging
- Currently turned off
[.column]
import pandas as pd
from py6.etl.redshift.tools import table_helpers
from py6.etl.redshift.tools import kafka_to_redshift
from py6.etl.redshift.processors import ingester_processing
TABLE_NAME = table_helpers.infer_table_name(__file__)
PLATFORM_PROCESSOR = ingester_processing.ProsperDeclineIngestionProcessor()
def process(messages):
# type: (kafka_to_redshift.ProcessorDataType) -> pd.DataFrame
batch_processor = ingester_processing.DeclineBatchProcessor(PLATFORM_PROCESSOR, TABLE_NAME)
return batch_processor.process(messages)
if __name__ == "__main__":
kafka_to_redshift.Loader.parse_args(
topic=PLATFORM_PROCESSOR.kafka_topic,
table=TABLE_NAME,
processor=process,
cluster="production",
db="prod").run()Kafka Hypothesis Generator
Intro to Hypothesis
[.column]
- Think of a normal unit test as being something like the following:
- Set up some data.
- Perform some operations on the data.
- Assert something about the result.
- Hypothesis lets you write tests which instead look like this:
- For all data matching some specification.
- Perform some operations on the data.
- Assert something about the result.
- (From hypothesis docs)
[.column]
def encode(input_string):
count = 1
prev = ""
lst = []
for character in input_string:
if character != prev:
if prev:
entry = (prev, count)
lst.append(entry)
count = 1
prev = character
else:
count += 1
entry = (character, count)
lst.append(entry)
return lst
def decode(lst):
q = ""
for character, count in lst:
q += character * count
return qKafka Hypothesis Generator
Intro to Hypothesis
[.column]
from hypothesis import given
from hypothesis.strategies import text
@given(text())
def test_decode_inverts_encode(s):
assert decode(encode(s)) == s[.column]
Falsifying example: test_decode_inverts_encode(s='')
UnboundLocalError: local variable 'character' referenced before assignmentKafka Hypothesis Generator
Setup
- To test if the decline ingester port worked properly, wanted to use Hypothesis to fuzzy test the ingester script with well-formatted Protobuf messages
- Create a general strategy that can generate messages for Hypothesis testing from a given Kafka topic
Kafka Hypothesis Generator
Demo
Kafka Hypothesis Generator
Sample usage
from py6.etl.redshift.loaders.prosper import my_etl_module
from py6.testing import kafka_to_redshift_helpers
sample_message_path = "path/to/sample/message.json"
Your sample message JSON should be formatted like this:
# [
# {
# "timestamp": 0.001,
# "value": google.protobuf.json_format(kafka_message_1)
# },
# {
# "timestamp": 0.002,
# "value": google.protobuf.json_format(kafka_message_2)
# },
# ...
# ]
#
@kafka_to_redshift_helpers.unit_test_loader(my_etl_module.TOPIC, sample_message_path)
def test_my_loader(processor_input):
df = my_etl_module.processor(processor_input)
# User needs to write assertions about the final dataframe
assert len(df[df['interest_rate'] > 0]) == 0 # bogus example assertionKafka Hypothesis Generator
Next steps
- Doesn’t currently work on Prosper topic - need to make an effective generator for TU attributes
- Can use hypothesis testing for testing functions involving Pandas dataframes as well
- @ezzeri Some processors in
dim_loanare good candidates
- @ezzeri Some processors in
That’s it!
Thank you all!
uid: 202005291000 tags: #theorem #presentation
