DynamoDB
Differences Between Relational Data Design and NoSQL
Relational database systems (RDBMS) and NoSQL databases have different strengths and weaknesses:
In RDBMS, data can be queried flexibly, but queries are relatively expensive and don’t scale well in high-traffic situations (see First Steps for Modeling Relational Data in DynamoDB).
In a NoSQL database such as DynamoDB, data can be queried efficiently in a limited number of ways, outside of which queries can be expensive and slow.
These differences make database design different between the two systems:
In RDBMS, you design for flexibility without worrying about implementation details or performance. Query optimization generally doesn’t affect schema design, but normalization is important.
In DynamoDB, you design your schema specifically to make the most common and important queries as fast and as inexpensive as possible. Your data structures are tailored to the specific requirements of your business use cases.
Approaching NoSQL Design
The first step in designing your DynamoDB application is to identify the specific query patterns that the system must satisfy.
In particular, it is important to understand three fundamental properties of your application’s access patterns before you begin:
Data size: Knowing how much data will be stored and requested at one time will help determine the most effective way to partition the data.
Data shape: Instead of reshaping data when a query is processed (as an RDBMS system does), a NoSQL database organizes data so that its shape in the database corresponds with what will be queried. This is a key factor in increasing speed and scalability.
Data velocity: DynamoDB scales by increasing the number of physical partitions that are available to process queries, and by efficiently distributing data across those partitions. Knowing in advance what the peak query loads might help determine how to partition data to best use I/O capacity.
After you identify specific query requirements, you can organize data according to general principles that govern performance:
Keep related data together. Research on routing-table optimization 20 years ago found that “locality of reference” was the single most important factor in speeding up response time: keeping related data together in one place. This is equally true in NoSQL systems today, where keeping related data in close proximity has a major impact on cost and performance. Instead of distributing related data items across multiple tables, you should keep related items in your NoSQL system as close together as possible.
As a general rule, you should maintain as few tables as possible in a DynamoDB application.
Exceptions are cases where high-volume time series data are involved, or datasets that have very different access patterns. A single table with inverted indexes can usually enable simple queries to create and retrieve the complex hierarchical data structures required by your application.
Use sort order. Related items can be grouped together and queried efficiently if their key design causes them to sort together. This is an important NoSQL design strategy.
Distribute queries. It is also important that a high volume of queries not be focused on one part of the database, where they can exceed I/O capacity. Instead, you should design data keys to distribute traffic evenly across partitions as much as possible, avoiding “hot spots.”
Use global secondary indexes. By creating specific global secondary indexes, you can enable different queries than your main table can support, and that are still fast and relatively inexpensive.
These general principles translate into some common design patterns that you can use to model data efficiently in DynamoDB.
GSIs in DynamoDB
Global Secondary Indices (GSIs) can be used to resolve queries in DynamoDB. To speed up queries on non-key attributes, you can create a global secondary index. A global secondary index contains a selection of attributes from the base table, but they are organized by a primary key that is different from that of the table. The index key does not need to have any of the key attributes from the table. It doesn’t even need to have the same key schema as a table. Because of this, in the GSI, records can be non-unique.
In this case, we can use the vendor code as the partition key, and the sort key as the weekend day.
Problem is the sort key is a date range, drawback of dynamoDB is that it cannot do some sort of range search.
You can’t do query of that kind. DynamoDB is sharded across many nodes by hash key, so doing a query without hash key (on all hash keys) is essentially a full scan. A hack for your case would be to have a hash key with only one value for the whole table, but this is fundamentally wrong because you loose all the pros of using DynamoDB. See hot hash key issue for more info: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html
Maybe we can do some sort of binary search? But in this data, weekend day is for 1 week, month data, might have to do 4 keys In this case, dynamodb might not be the best
Query in DynamoDB
Class called dynamodb query expression, you can define which attribute you want to use for the query You can define a condition for the expression, define larger than, equals, even other conditions, or some other math condition
When not to use DynamoDB
When not to use DynamoDB:
When multi-item or cross table transactions are required When complex queries and joins are required When real-time analytics on historic data is required
Should I use DynamoDB?
DynamoDB
uid: DynamoDB tags: #amazon
Talk With Trisha
Just like her process of exploring stuff, how she was scared to admit it but it’s a lot more relieving to be comfortable with herself, stuff like that Yeah she actually already told her like a week ago So she already knew But she hasn’t told daddy yet So we talked about that too And then about family and relationships and priorities and also a continuation of the career talk we had before about like Success in life and what’s actually important We talked about us too Dude my family really really really likes you And my sister really really wants to get to know you better She likes you a lot too Not too but like Yeah she really likes you And they’re not worried about anything like The time will come for you guys to get to know each other better like They were saying maybe in a couple years or something we could go on some kind of trip where it would be like physical time spent together And also like how trust doesn’t really form until ppl have a disagreement but then get past it and move on And you guys haven’t really had the opportunity to do that yet so it’s fine So yeah we talked about us too And we talked about my dad And friends and you won’t find people directly on your wavelength like we talked about different examples of instances with even people that we were close to that we didn’t agree with some things that they said or did umm We talked about this Indian actor who committed suicide sushant Singh rajput At this point I’m listing out everything even if it wasn’t too relevant so I can use this as a reference for the note I’m going to write about it LOL
- It’s awesome that she has a friend group that’s present with her along the process, and she can talk to them and bounce ideas off of them and know that they’ll be accepting and understand. That’s something Mulan was really happy about as well.
uid: 202006270159 tags: #talks
Long-Term Goals
I’m going to put these into Complice, but this works for GTD too.
(These are copy-pasted directly from Complice currently. Definitely subject to change.)
- Be organized.
Be organized and precise with the activities I do and information I process. Have a capable enough system that I never forget knowledge I gained, skills I learned, or experiences I had at any point in time.
- Do my best on all my commitments.
I don’t want to be half-hearted with the things that I’m choosing to spend my time on - I want to work on things wholly and earnestly, and spend as much attention and time as I can on the things that I decide matter to me in that moment.
- Be practically independent.
I want to be able to live a healthy, fulfilling life independently if need be. This means learning and improving myself in a few areas — driving, cooking, finances, odd jobs around the house, and other aspects that come with being independent.
- Grow grow grow.
Pretty obvious, and natural for me. Keep learning, growing, developing, trying new things in the things I’m interested in (and a few things I’m not too interested in as well).
- Develop my social network.
I want to expand my network, both professionally (so I have a number of people I can reach out to in case I need career help or advice), and socially. I want to actively seek out meaningful experiences with other people, and find genuine people that I can trust and make deep connections with.
- Be a good son/brother/boyfriend/friend
Be a better person for the people I truly care about.
- Contribute to the world 202006140156
Give back to the world from which I’ve taken so much. Both in ways I’m comfortable with and ways I’m currently not (donating blood, protests, going outside of my comfort zone in ways that make an impact)
- Be more forward-thinking
Stop myself from falling down patterns that I know aren’t good for me. Tackle big picture problems related to my future head-on. Engage in regular review to make sure I’m living up to the standards I have for myself.
- Be more disciplined
Have the motivation and self-discipline to hold myself accountable to the things that I commit to, and tasks that I know I need to do to accomplish the rest of my goals.
- Be physically healthy
Pretty obvious. Currently care more about cardiovascular health and physical fitness than getting buff.
uid: 202006221637 tags: #living-well #productivity
Thoughts on the protests, aggregated by Mulan
Facts
- The people killed 1,099 people in 2019. 24% of victims were black, while only 13% of the US population is black.
- Violent offenders who are black: 22.7% Violent offenders who are LatinX: 14.4% Violent offenders who are white: 43.8%
Privilege-questioning thoughts
- People seem to forget that George Floyd died because people were trying to follow the law and negotiate peacefully.
- When as a white/non-black person, you say that “violence is not the answer”, you are saying “peaceful protest and negotiation” are the answer. Saying that denies (or ignores) the simple reality that marginalized groups often aren’t heard.
- It is not your choice to determine how an oppressed group protests.
- The current riots and violence are a result of countless peaceful protests that haven’t worked and anger towards the racist American justice system that continues to let black people be murdered unjustly.
- Riots are not the problem. Riots are a symptom of the problem.
- Violent protests have consequences. People will die, people will go to jail, people will lose everything they have. How far does someone have to be pushed to risk it all?
- Yes, riots feel uncomfortable because they are uncomfortable. Racism of also uncomfortable.
- If it’s difficult for you to understand why people resort to violence, it probably means your privilege has protected you from being put in a situation where you feel you have no other choice.
- Implying there are other ways minimizes the frustration and provides no answers, or solutions. It just points, with privilege, to do things a different way that doesn’t disturb your comfort.
General Thoughts about the Protests/situation
- Sometimes change is disruptive. Peaceful protest hasn’t worked, which is why they are resorting to riots, but some people are taking advantage of the situation, and just causing destruction. It’s not just black people looting.
- It’s important to note that
- It’s not just black people looting: most videos show white people and officers instigating violence
- Black and indigenous people have lost eons worth of ancestors and knowledge due to white greed and haven’t been close to repaid. They are taking what’s owed to them.
We live in a world where trained cops can panic and act on impulse, but untrained civilians must remain calm with a gun in their face.
How to be an ally, as an Asian-American
- Recognize your privileges
- The black experience is different from ours. We don’t face the same severe multi-faceted discrimination as them, and we never will
- Utilize your privileges to leverage support for movements for black causes
- Recognize that a lot of our freedoms and rights are credited to the movements and strides made by African-American activists during the Civil Rights era
- Support black-owned businesses
- Call out your family and friends on their bullshit
- Understand that the government has been using us as a scapegoat for their mistreatment of black people (i.e. the model minority myth)
- Our privileges are a mirage given to us by the unjust powers as an effort to use us as a racial wedge
People ask “Can’t they demand justice without burning stuff down?” Most of these protest started out peacefully until the police escalated things as they usually do. It’s also the non-black people who don’t actually care, and just want to add to the flames.
George Floyd couldn’t breathe. Eric Garner couldn’t breathe. Breonna Taylor, Amaud Arbery, Tony McDade. Botham Jean, Atatiana Jefferson, Stephon Clark, Philando Castille, Trayvon Martin, Michael Brown, Tamir Rice, Jonathan Farrell, Renisha McBride, Jordan Edwards, Jordan Davis, Aiyana Jones, the Charleston 9, Sean Bell, Oscar Grant, Sandra Bland, Corey Jones, John Crawford, Terrence Crutcher, Clifford Glober, Claude Reese, Randy Evans, Yvonne Smallwood, Amadou Diallo, Walter Scott, Freddie Gray, and countless others, can’t breathe.
There are not my thoughts — In fact, I don’t completely agree with all of these, especially the one seems to justify that centuries of violence justifies more violence. But I think these notes are interesting as a reminder to me, especially so that I can gradually phase off my own thoughts that aren’t very ideal 202006020157.
uid: 202006011540 tags: #literature
Just to add to this (and off the bat I will say I’m pretty much drawing on Francine Hirsch’s Empire of Nations): late Tsarist-era/early-Soviet ethnographers tended to be influenced by then-fashionable German concepts of Voelkisch-ness (which saw ethnic groups in a semi-scientific light), while communist thinkers themselves were more influenced by French concepts of nationality (ie, that nations were the result of a collective conscious identity).
As noted above, Soviet anthropologists did engage in physical anthropology, notably study of blood types. Nevertheless, debates in the 1920s over how to define ethnic groups largely rejected using any sort of physiological traits or blood origin, but rather focused on group consciousness (although, for example, in the 1926 Census, if a respondent didn’t have a nationality to self-identify with, the census takers could prompt them with “objective” determinants like kinship, region, religion or language).
Soviet anthropology and ethnography in the 1920s and 1930s, adapted fairly easily to the new regime, and found general Marxist concepts of stages of development amenable to how it presented different ethnic groups: ie, if culture was superstructure to an economic base, and if a people’s economy was determined by environmental factors, then the perceived “primitiveness” of, say, Siberian pastoralists was explainable in terms of environment and economy, rather than race.
Soviet anthropology was heavily influenced by Edward Burnett Tylor and his stages of development, notably his concept of “survivals” (which were basically customs or habits from earlier stages of development that were carried through to later stages of development by force of habit). These “survivals” (perezhitki in Russian) became a particular target for explaining public health problems that, in the West, would be explained by eugenics at the time - so for example, veiling in Central Asia was seen as a cultural practice that promoted the spread of syphilis and the weakening of health of children.
The rejection of “race science” by Soviet anthropology became explicit after 1930, and in no small part for international political reasons: it was championed by Nazi Germany, and so the USSR, for political and ideological reasons, put a lot of effort into disproving biological determinism, hierarchies of race, and the idea of racial “degeneracy”. It should be noted that this actually wasn’t a debate from afar: the Soviets and Germans conducted joint anthropological expeditions in the USSR from 1920 to 1937, so for instance there were expeditions to the Chuvash and to the Soviet Far East where the German anthropologists examined physical characteristics and the Soviets examined social and cultural characteristics based on economic conditions. The Soviets also attempted to work out theories of “non-racist race science”, largely based on the work of Franz Boas, that saw group physical characteristics as caused by isolation, and migration/contact with other groups as a means of undoing these characteristics.
Of course, it’s worth noting that sometimes these efforts went to the other extreme. There was interest in proving the ideas of Lamarck, namely in the concept of “acquired characteristics” - ie that people could change their physical characteristics and pass on those new characteristics to their offspring. This fit very well into the concept of Homo sovieticus, and the ability of the Stalinist revolution to change the culture and very physicality of Soviet people. Of course, the absolute extreme of this would be Trofim Lysenko and his charlatin “vernalism”. Lysenko specifically worked with botany (not anthropology), but he was favored by Stalin for developing these Lamarckian views, and because of his political favor and his persecution of rivals, he essentially destroyed Soviet biology for a generation.
A final note: outside of academia, there were, especially in Central Asia and Siberia, some vague ideas of race, but these tended to be expressed in terms of “European” and “Asian” or “Mongolic”, and of course where Europe ends and Asia begins in Russia is a whole separate topic in itself. For what its worth, these were largely more cultural than racial concepts, so in Central Asia, there was very much a distinction between local Asians and “Europeans”, but “European” included Russians, other Slavs, Ashkenazi Jews, and Tatars.
Overall, the Soviet Union was far more interested in developing and cataloging nationality (and the stages of development those nationalities were in) than race.
Professor Roman notes that in this case the other factory workers themselves acted as judge, jury, and executioner (rather, deporter) of these two men, which was an idiosyncratic way of doling out Soviet justice as well that was part of the entire anti-racist schtick the USSR was playing up at the time. Please don’t interpret that moniker as some belittlement of anti-racism, but I think in the case of the Soviet Union, to say the government was acting in good faith (as opposed to calculated propaganda) in this case would be dishonest at the least and outright insulting to people who experienced inarguable racial discrimination in the USSR. Other works I’ve read by Roman really, really downplay the amount of actual racism that occurred in the Soviet Union and really, really play up the admittedly forward-thinking posture the Soviet Union took towards African Americans in the 1920s and 1930s.
For example in Opposing Jim Crow (2012) she writes:
The 1928 Comintern decree was also consistent with Soviet nationality policy, which afforded the officially sponsored non-Russian nationalities of the USSR the nominal right to national self-determination. Besides encouraging their cultural development, Soviet leaders established an ethnicity-based affirmative action system that privileged non-Russians over ethnic Russians in terms of hiring, admissions, and promotions. [3]
Factually true? Absolutely. Missing important, if not critical, contextualization? Also, absolutely.
Professor Roman brings up an important point though, which are the many programs present in the Soviet Union which we would today call ‘Affirmative Action.’ Indeed, she repeatedly refers to the Soviet Union as the Affirmative Action Empire– which to be frank, I can’t tell if she’s saying in a cheeky way or not. Besides the idea of calling the Soviet Union an empire that I talk about in this answer, someone so openly anti-imperialist in every imaginable capacity using it as a potential positive descriptor is a little bizarre on the face of it. But on to these programs– besides the practice of promoting and positively selecting for non-Russians on occasion, the early Bolsheviks implemented a number of localization programs under the aegis of wide-sweeping attempts to ‘de-Russify’ what had formerly been the Russian Empire. These were called in bulk korenizatsia which can be translated roughly to localization, though really it’d be closer in a literal sense to the non-existent word ‘root-ification.’ Root here being used the way English speakers describe the root cause of something or the root of a word etymologically.
Korenizatsia is championed as an example of explicit Soviet anti-racism and was indeed, fairly innovative in the context of turn of the 20th century Europe when nationalism was on the rise just about everywhere else. Most recently (that I’m aware of), a group of Finnish historians (as well as one German historian) wrote about this and other processes in a 2018 work called The Fall of an Empire, the Birth of a Nation which studies the evolution of racial identities from the Russian Imperial period to the modern day Russian Federation. They make a really profound point which is that the Russian Empire was unique in that nearly all of its so-called colonial holdings were along its contiguous borders. This created a special need for the inheritor of this colonial-ish mantle to address more directly issues of national and ethnic sovereignty than say, the British Empire– the majority of whose subjects lived in lands far away from ol’ Blighty and primarily were of a drastically different skin color. The authors go so far as to say the following:
What is […] missing [in the political discourse of the Russian nation] is the open acknowledgement of Russian national identity as multiethnic and multicultural. (see Chapter 2, Origin and Power…)
As much as the initial salvo I provided you with concerning blatantly and unarguably racist practices in the Soviet Union were just that, as we’ve seen above there were also far-reaching, legitimate, and good faith attempts to deconstruct racism (and indeed nationalism) as a concept completely. That deserves positive accolades the same way the latter days sins of Stalin deserve unmitigated condemnation. The Soviet Union was a paradox in many, many ways– as you see here, its treatment of and relationship to ethnic minorities is just one in a long list of reasons why.
In conclusion (what a tacky ending right?), we have a strong idea of how both racism and anti-racism presented themselves with respect to the Soviet government, and it’s important to note that many of the more egregious actions I mention above largely, though not exclusively, occurred before or during Stalin’s reign (which spanned nearly three decades– four if we include Lenin’s rule as well– nearly, or more than, half of the lifespan of the entire Soviet project), so although partly his doing, it’s unfair to entirely dismiss these occurrences as the machinations of one bad apple. There was a systemic element to Soviet racism (as well as anti-racism though too). I think that is the most important issue raised by your question because exploring individual-to-individual racism is going to immediately start running into all those issues I mentioned above, it’s just such a vast question that I think you’ll just have to be satisfied with a rather unsatisfying, ‘yes, there was racism in the USSR.’ How were ethnic minorities treated? Sometimes good, sometimes okay, sometimes really, really not okay. Next article: 202005151927 Part 4 - Soviet union