In this document I am going through a couple of commands and use cases using R.
This follows more or less Exercise 2.2.

In terms of R, this covers most of what you need for the graded part of this course.
It also largely covers what would be needed for the group coursework (although you are of course welcome to go beyond).

I hope you will appreciate that these are actually only a few commands and concepts.
R can be confusing because it can do so many things. But think of it like learning a language, which means:

- It takes a while to get used to

- You have to practice regularly

- You cannot expect to read and write abstract poetry or a novel that will be nominated for the Nobel Prize after a week

- However, you should be able to use a couple of words and expressions right away and go and use this as much as possible while discovering new words and expressions here and there.

- Also: you will be able to use those expressions successfully without necessarily understanding the grammatical logic or etymology behind them. But as you keep on at you might find out these
things at some point.



What I am trying to do in this document and in the course overall is to provide you a couple of R words and expressions for you to play around with. Playing means:

- you try the same commands with different data (e.g. the data for your group project)

- you try change things here and there (change how a figure looks slighlty, change how a variable is defined)

- you think of something you might want to do (construct a new variable based on exisitng ones, create a figure, combine and clean some unusual data). You see if you can do it by consulting the help functions and the internet. If you don't manage you can ask your teachers (us) for help.


So what are the basic expressions in R that you want to start your journey to Rland with; i.e. the equivalent of ordering a coffee and a croissant?


# Getting data
You will want to start with getting data. Let's look at the data from Exercise 2.2 however it will be very similar if you were to use some other data. 


```{r}

covid=read.csv("https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv")

```


i.e. you use the `read.csv` command to load the dataset into the memory as a data frame


# Data manipulation and cleaning

Data comes rarely in a way such that you can start analysis right away. So you need to know some commands that help you with things like removing some observations or creating new variables.


## Create new variables from existing and filter data

```{r}
library(dplyr)

# keep the data point for  the end of July 2020
covid_july=covid %>%  
  filter(date=="2020-07-31") 


# create various ratios
covid_july = covid_july %>%
                 mutate(
                    deathsOcases=deaths/cases,
                    deaths_sh=deaths/sum(deaths),
                    meandeaths=mean(deaths),
                    cases_sh=cases/sum(cases))

```






Part of data manipulation is that you look at your data. We already saw the summary command

```{r}
covid_july %>% summary()


```


With that we see that the maximum death rate is nearly 9%. But which state was that in?
You can use the filter command again to work it out:

```{r}

covid_july %>% filter(deathsOcases>0.088)

covid_july=covid_july %>% arrange(deathsOcases)
covid_july=covid_july %>% arrange(-deathsOcases)


```


## Compute summary statistics or aggregate data

For instance work out the total death and case figures from state level ones

```{r}
# now we aggregate to daily level
usdaily=covid %>% group_by(date) %>% 
        summarise(deaths=sum(deaths), 
                  cases=sum(cases))
                 
head(usdaily)
#### make lag and lead example


```

What if you want something other than the sum of a variable? There loads of other functions.
You can start by using help function (by typing `?summarise()` in the console) or read about the summarise command online (just google `dplyr summarise` which would lead you for instance to [this](https://dplyr.tidyverse.org/reference/summarise.html))


## Combining data
First you need another dataframe to combine 
```{r}
pop=read.csv("https://mondpanther.github.io/datastorieshub/data/populationdata.csv")

```


You can use the `merge` command which is an older command. A newer version of the same idea
are the join commands (As I mentioned: the difficulty with R is that there are often several different commands that do more or less the same)



```{r,echo=TRUE}
covid_julyb=covid_july %>% left_join(pop,by="state")

```




## Calculations that involve values from different rows

The `mutate` command examples we have seen so far create new variables by combining values from existing variables that are in the same row of a dataframe (e.g. when computing `deathsOcases`). We also have seen examples where we use values that refer to the same line as well as calculations that apply to the full dataframe columns (e.g. when computing the share variables such as `deaths_sh=deaths/sum(deaths)`). Sometimes we need to compute new values that are based on values in other rows but not necessarily all values as in `sum(deaths)`. A good example and use case of that emerges from the covid figures used earlier. Note that the dataset we are using report cumulative figures of deaths and cases for the whole pandemic. What if instead we wantd to examine the daily figures instead? For that we would have to look at the change in cumulative numbers for a given day (i.e. the cumulative figure today minus that of yesterday for all days)

To calculate that on the `usdaily` dataframe we can use the `lag()` function within the `mutate()` function:

```{r}
usdaily=usdaily %>% mutate(Dcases=cases-lag(cases,1),
                           Ddeaths=deaths-lag(deaths,1))

```

i.e. we subtract the value of the `cases` and `death` variables in a given row with the value in the previous row (make sure the table is sorted in the right way; i.e. in this case by date).
Also note that you that the second parameter controls the lag; i.e. if you wanted to subtract the value 2 days back you would have to put a 2 instead of a 1. There is also a function that allows you to refer to values in rows further down in the dataframe; it's called `lead()`.

Note that you can combine that with the `group_by` command. For instance, in the original covid dataframe you might want to get daily changes by US state:

```{r}
covid=covid %>% arrange(state,date) %>% 
  group_by(state) %>%  
  mutate(Dcases=cases-lag(cases,1),
                           Ddeaths=deaths-lag(deaths,1))

```

Compare this with using `lag` command without `group_by`:


```{r}
xcovid=covid %>% arrange(state,date) %>% 
  group_by(state) %>%  
  mutate(Dcases=cases-lag(cases,1),
                           Ddeaths=deaths-lag(deaths,1))

```

Can you see the difference `covid` and `xcovid`? Can you explain what is wrong in the `xcovid` way of computing daily changes?

# Data visualisation

Let's run through the ggplot command once more.....as I said: it's a bit of swiss army
knife of plotting


## Scatter plot

However first: with more data comes more data manipulation; e.g. cases per capita....

```{r}
covid_julyb=covid_julyb %>%  
  mutate(casesOpop000=cases/pop*1000)
```


Now how about a scatter plot of cases per capita and population density
(measured in people per square mile)?

```{r}
library(ggplot2)


p=ggplot(covid_julyb, 
         aes(x=density, y=casesOpop000 )) +  
         geom_point() 
p

p + geom_smooth(method=lm)


# Dropping an outlier

ggplot(covid_julyb %>% filter(density<6000), aes(x=density, y=casesOpop000 ))+ 
      geom_point() + 
      geom_smooth(method=lm)


# Less clutter

#newfr=covid_julyb %>% filter(density<6000)

ggplot(covid_julyb %>% filter(density<6000), aes(x=density, y=casesOpop000 ))+ 
      geom_point() + 
      geom_smooth(method=lm)+theme_minimal()


```

## Time series plot

```{r}
library(ggplot2)

usdailyx=usdaily %>% mutate(date=as.Date(date),
  deathsOcases=deaths/cases)

ggplot(usdailyx, aes(x=date, y=deathsOcases)) +
      #geom_point() +
      geom_line() +
      scale_x_date(date_breaks="2 month",
                   date_labels = "%b")

```

## Density plot
```{r}
covid_julyb %>% ggplot(aes(x=casesOpop000))+geom_density()
```

Note, that what we put in the aesthetic depends on the plot type. A density plot only needs one variable.
# Fitting an econometric model

The simplest case is the linear model which we can fit using the OLS algorithm. The
`lm()` command is your tool for that.


```{r}
df=covid_julyb  %>% filter(density<6000& !is.na(casesOpop000)) 

reg=lm(casesOpop000~density,df)

summary(reg)

reg %>% summary()



```



Let's examine the residuals for this regression



```{r}

df=df %>%  mutate(resids=reg$residuals)
ggplot(df, aes(x=density, y=resids ))+ 
      geom_point() + 
      geom_smooth(method=lm)+theme_minimal()


cor(df %>% select(density,resids,casesOpop000))

```

Note there is nocorrelation between residuals and density.


