DataStory

[SQL] 2. LOGIC

효효캉 — Wed, 27 Jan 2021 15:18:59 +0900

SELECT * FROM benn.college_football_players

benn.college_football_players database

1. CASE

SELECT 절에 사용
CASE WHEN __[case 1]__ THEN _[case 2]__ ELSE(optional) END
WHEN과 THEN 사이에 WHERE 같은 conditional statment 사용가능
- multiple conditional statment 사용 시 AND , OR 사용
WHEN, ELSE 여러번 사용 가능

SELECT player_name,
       year,
       CASE WHEN year = 'SR' THEN 'yes'
            ELSE NULL END AS is_a_senior
  FROM benn.college_football_players

** ELSE: optional. provides a way to capture values not specified in WHEN/THEN statemennt
** CASE + WHEN ~~~ THEN is necessary

SELECT player_name,
       year,
       CASE WHEN year = 'SR' THEN 'yes'
            ELSE 'no' END AS is_a_senior -- SR이 아닐경우 NULL이 아니라 'no'로 표시
  FROM benn.college_football_players

Q: Write a query that includes a column that is flagged "yes" when a player is from California, and sort the results with those players first.
A: SELECT player_name,
       state,
       CASE WHEN state = 'CA' THEN 'yes'
       ELSE NULL END AS is_from_Cali
  FROM benn.college_football_players
  ORDER BY 3

1.2. Adding multiple conditions to CASE
SELECT player_name,
weight,
CASE WHEN weight > 250 THEN 'over 250'
WHEN weight > 200 THEN '201-250'
WHEN weight > 175 THEN '176-200'
ELSE '175 or under' END AS weight_group
  FROM benn.college_football_players
-- better way to do this is the following:
SELECT player_name,
       weight,
       CASE WHEN weight > 250 THEN 'over 250'
            WHEN weight > 200 AND weight <=250 THEN '201-250'
            WHEN weight > 175 AND weight <= 200 THEN '176-200'
            ELSE '175 or under' END AS weight_group
  FROM benn.college_football_players

Q: Write a query that includes players' names and a column that classifies them into four categories based on height. Keep in mind that the answer we provide is only one of many possible answers, since you could divide players' heights in many ways.
A:
SELECT player_name,
       height,
       CASE WHEN height >80 THEN 'over 80'
            WHEN height >70 AND height <=80 THEN '71-80'
            WHEN height >60 AND height <=70 THEN '61-70'
            WHEN height >50 AND height <=60 THEN '51-60'
            ELSE 'under 50' END AS height_group
  FROM benn.college_football_players
**WHEN과 THEN 사이에 conditional operator 추가가능; AND OR

SELECT player_name,
       CASE WHEN year = 'FR' AND position = 'WR' THEN 'frosh_wr'
            ELSE NULL END AS sample_case_statement
  FROM benn.college_football_players

SELECT *,
       CASE WHEN year='JR' THEN 'junior'
            WHEN year ='SR' THEN 'senior'
            ELSE NULL END AS junior_or_senior
  FROM benn.college_football_players

Q: Write a query that selects all columns from benn.college_football_players and adds an additional column that displays the player's name if that player is a junior or senior.

A:
SELECT *,
       CASE WHEN year IN ('JR', 'SR') THEN player_name
       ELSE NULL END AS upperclass_player_name
  FROM benn.college_football_players

SELECT CASE WHEN year = 'FR' THEN 'FR'
            ELSE 'Not FR' END AS year_group,
            COUNT(1) AS count
  FROM benn.college_football_players
GROUP BY CASE WHEN year = 'FR' THEN 'FR'
               ELSE 'Not FR' END
-- COUNT(1) same as COUNT(*)

SELECT COUNT(1) AS fr_count
  FROM benn.college_football_players
  WHERE year ='FR' -- why not use this? WHERE only allows you to count one condition

SELECT CASE WHEN year = 'FR' THEN 'FR'
            WHEN year = 'SO' THEN 'SO'
            WHEN year = 'JR' THEN 'JR'
            WHEN year = 'SR' THEN 'SR'
            ELSE 'No Year Data' END AS year_group,
            COUNT(1) AS count
  FROM benn.college_football_players
  GROUP BY 1 -- or year_group
            -- or BY CASE WHEN year = ~

Q: Write a query that counts the number of 300lb+ players for each of the following regions: West Coast (CA, OR, WA), Texas, and Other (Everywhere else).
A:
SELECT CASE WHEN state IN ('CA', 'OR', 'WA') THEN 'West Coast'
            WHEN state = 'TX' THEN 'Texas'
            ELSE 'Other' END AS Area_Category,
            COUNT(1) as count
  FROM benn.college_football_players
  WHERE weight >= 300
  GROUP BY Area_Category -- or GROUP BY 1

Q: Write a query that calculates combined weight of all underclass players(FR/SO) in California as well as the combined weight of all upperclass players (JR/SR) in California
A:
SELECT CASE WHEN year IN ('FR', 'SO') THEN 'underclass players'
            WHEN year IN ('JR', 'SR') THEN 'uppderclass players'
            END AS class_division,
            SUM(weight) AS sum_weight
  FROM benn.college_football_players
  WHERE state = 'CA'
  GROUP BY class_division --or 1

SELECT CASE WHEN year = 'FR' THEN 'FR'
            WHEN year = 'SO' THEN 'SO'
            WHEN year = 'JR' THEN 'JR'
            WHEN year = 'SR' THEN 'SR'
            ELSE 'No Year Data' END AS year_group,
            COUNT(1) AS count
  FROM benn.college_football_players
GROUP BY 1

-- horizontal
SELECT COUNT(CASE WHEN year = 'FR' THEN 1 ELSE NULL END) AS fr_count,
       COUNT(CASE WHEN year = 'SO' THEN 1 ELSE NULL END) AS so_count,
       COUNT(CASE WHEN year = 'JR' THEN 1 ELSE NULL END) AS jr_count,
       COUNT(CASE WHEN year = 'SR' THEN 1 ELSE NULL END) AS sr_count
  FROM benn.college_football_players

Q: Write a query that displays the number of players in each state, with FR, SO, JR, and SR players in separate columns and another column for the total number of players. Order results such that states with the most players come first.
A:
SELECT COUNT(CASE WHEN year = 'FR' THEN 1 ELSE NULL END) AS fr_count,
       COUNT(CASE WHEN year = 'SO' THEN 1 ELSE NULL END) AS so_count,
       COUNT(CASE WHEN year = 'JR' THEN 1 ELSE NULL END) AS jr_count,
       COUNT(CASE WHEN year ='SR' THEN 1 ELSE NULL END) AS sr_count,
       COUNT(1) AS total_players,
       state
  FROM benn.college_football_players
  GROUP BY state
  ORDER BY total_players DESC

Q: Write a query that shows the number of players at schools with names that start with A through M, and the number at schools with names starting with N - Z.
A:
SELECT CASE WHEN school_name < 'n' THEN 'A-M'
            WHEN school_name >= 'n' THEN 'N-Z'
            ELSE NULL END AS school_name_group,
       COUNT(1) AS players
  FROM benn.college_football_players
GROUP BY school_name_group --or 1

[SQL] 1. Aggregate Functions

효효캉 — Wed, 27 Jan 2021 15:02:39 +0900

We are going to use the following database:

SELECT * FROM tutorial.aapl_historical_stock_price

tutorial.aapl_historical_stock 데이터베이스

1. Aggregate functions in SQL

SELECT / HAVING 에 사용

COUNT: 특정 칼럼에 몇개의 행이 있는지 count
SUM: 특정 칼럼에 있는 값의 총합
MIN & MAX: 특정 칼럼의 가장 작은(earliest date, smallest, closest to "A") / 큰 값
AVG: calculates average of group of selected values
Basic Tutorial 에서 말한 것들은 across rows. 여기는 칼럼 전체에 관한 계산

2. Additional Basics of SQL

GROUP BY : COUNT, AVG, SUM: aggregate across entire table. What if you want to aggregate only part of the table? Use groupby
ORDER BY
HAVING : filter the groupby result. WHERE clause doesn't work bc it doesn't allow you to filter aggregate columns

1.1 Counting all rows
SELECT COUNT(*)
FROM tutorial.aapl_historical_stock_price
-- returns total number of rows

1.2 Counting individual columns
SELECT COUNT(high)
  FROM tutorial.aapl_historical_stock_price
  -- counts all the rows except those with NULL

Q: Write a query to count the number of non-null rows in the low column
SELECT COUNT(low)AS low
  FROM tutorial.aapl_historical_stock_price
-- If you don't specify the column name, result will be just shown as COUNT

1.3 Counting non-numerical columns
SELECT COUNT(date) AS count_of_date
  FROM tutorial.aapl_historical_stock_price
SELECT COUNT(date) AS "Count Of Date"
  FROM tutorial.aapl_historical_stock_price
-- if you need to use spaces, use double quotes

Q: write a query that determines counts of every single column
Which column has the most null values?
SELECT COUNT(year) AS year,
       COUNT(month) AS month,
       COUNT(open) AS open,
       COUNT(high) AS high,
       COUNT(low) AS low,
       COUNT(close) AS close,
       COUNT(volume) AS volume
  FROM tutorial.aapl_historical_stock_price

2. SUM

- COUNT 는 그냥 행의 갯수를 계산
- SUM 은 실제 값에 관한 계산
**Aggregate vertically! If you want to calculate across rows, use simple arithmetic

SELECT SUM(volume)
  FROM tutorial.aapl_historical_stock_price

Q: Write a query to calculate the average opening price
SELECT SUM(open)/COUNT(open) AS "Average Opening Price"
  FROM tutorial.aapl_historical_stock_price

3. MIN/MAX

SELECT MIN(volume) AS min_volume,
       MAX(volume) AS max_volume
  FROM tutorial.aapl_historical_stock_price

Q: What ws Apple's lowest stock price?
SELECT MIN(low)
  FROM tutorial.aapl_historical_stock_price

Q: Highest single day increase in stock price?
SELECT MAX(close-open)
  FROM tutorial.aapl_historical_stock_price

4. AVG

**Limitations: ONLY can use for numerical columns
IGNORES NULLS completely -> You cannot treat null value as 0

SELECT AVG(high)
  FROM tutorial.aapl_historical_stock_price
  WHERE high IS NOT NULL
SELECT AVG(high)
  FROM tutorial.aapl_historical_stock_price
-- 2 queries above have the same result

Q: Write a query that calcualtes the average daily trade volumns for Apple Stock
SELECT AVG(volume) as avg_volume
FROM tutorial.aapl_historical_stock_price

5. GROUP BY

5.1 SQL GROUP BY clause
SELECT year,
       month,
       COUNT(*) AS count
  FROM tutorial.aapl_historical_stock_price
GROUP BY year, month

Q: Calculate the total number of shares traded each month. Order results chronologivally.
SELECT year,
      month,
      SUM(volume) AS volume_sum
  FROM tutorial.aapl_historical_stock_price
  GROUP BY year, month
  ORDER BY year, month

5.2 GROUP BY column numbers
SELECT year, month, COUNT(*) AS count
  FROM tutorial.aapl_historical_stock_price
  GROUP BY 1,2

5.3 Using GROUP BY with ORDER BY
SELECT year, month, COUNT(*) AS count
  FROM tutorial.aapl_historical_stock_price
  GROUP BY year, month
  ORDER BY year, month
  -- ORDER BY month, year : totally different result

Q: Write a query to calculate the average daily price change in Apple stock, grouped by year
SELECT year, AVG(high - low) AS avg_daily_change
  FROM tutorial.aapl_historical_stock_price
  GROUP BY year
  ORDER BY year

Q: Write a query that calculates the lowest and highest prices that Apple stock achieved each month
SELECT year, month,
       MIN(low) AS min_price_monthly,
       MAX(high) AS max_price_monthly
  FROM tutorial.aapl_historical_stock_price
  GROUP BY month, year
  ORDER BY year, month

6. HAVING

SELECT year, month, MAX(high) AS month_high
  FROM tutorial.aapl_historical_stock_price
  GROUP BY year, month
  HAVING MAX(high) > 400
  ORDER BY year, month

[SQL] 0. Basic SQL

효효캉 — Thu, 21 Jan 2021 17:47:31 +0900

1. AS

2. (Filtering) Comparison Operators

3. (Filtering)Logical Operators

LIKE: 비슷한 값 매치 ; "column_name" LIKE 'Snoop%' -> Case sensitive
ILIKE: "column_name" ILIKE 'snoop%' -> not case sensitive 대소문자 상관없음!
IN: 포함하고 싶은 값의 리스트 설정
BETWEEN: certain range에 해당하는 rows 셀렉 -> HAS to be paired with AND operator
IS NULL: 주어진 칼럼에 데이터 없는 rows 셀렉
AND: 2가지 컨디션 모두 충족하는 행 선택
OR: either two conditions 컨디션 충족하는 행 선택
NOT:certain condition에 match하지 않는 행 선택

4. (Sorting )ORDER BY

Using AS

SELECT *
  FROM tutorial.us_housing_units
  LIMIT 5

SELECT west AS "West Region"
    FROM tutorial.us_housing_units
**only returns capital letters if covered with double quotes

Comparison Operators

= , <, >, <=, >=, <>

SELECT west AS West_Region,
south AS South_Region
FROM tutorial.us_housing_units

SELECT *
FROM tutorial.us_housing_units
WHERE west > 50

SELECT *
FROM tutorial.us_housing_units
WHERE south <= 20

SELECT *
FROM tutorial.us_housing_units
WHERE month_name = 'February'
** month_name is "n" or earlier

**"n"보다 큰걸로 설정하면 "na"부터 시작

SELECT *
FROM tutorial.us_housing_units
WHERE month_name < 'o'

Arithmetic in SQL

Using + - * / : ONLY IN SAME ROW
If you want to combine across diff rows, try aggregate functions

SELECT south + west + midwest + northeast AS Four_Regions
FROM tutorial.us_housing_units

SELECT *
FROM tutorial.us_housing_units
WHERE west > (midwest + northeast)

SELECT south/(south+west+midwest+northeast)*100 AS south_perc,
west/(south+west+midwest+northeast)*100 AS west_perc,
midwest/(south+west+midwest+northeast)*100 AS midwest_perc,
northeast/(south+west+midwest+northeast)*100 AS northeast_perc
FROM tutorial.us_housing_units
WHERE year >= 2000

Logical Operators

0.DB
SELECT * FROM tutorial.billboard_top_100_year_end
LIMIT 5

1. LIKE
double quote: used when 칼럼 이름 표시할때
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" LIKE 'Snoop%'

1.1. ILIKE
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" ILIKE 'snoop%'
  LIMIT 40

**underscore ; _ used for 특정 문자 대체할때
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE artist ILIKE 'Dr_ke'

Q: Write a query that returns all rows for which Ludacris was a member of the group
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" ILIKE '%Ludacris%'
Q: Write a query that returns all rows for which the first artist listed in the group has a name that begins with "DJ"
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" LIKE 'DJ%'

2. IN
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank IN (1,2,3)
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE artist IN ('Taylor Swift', 'Usher', 'Ludacris')
**Comparison Operator에서처럼 글자는 싱글 quote!

Q: Write a query that shows all of the entries for Elvis and M.C. Hammer
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" ILIKE '%Hammer%'
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" IN ('M.C. Hammer', 'Hammer', 'Elvis Presley')

3. BETWEEN
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank BETWEEN 5 AND 10

** 5랑 10 포함
** 위 쿼리는 아래 쿼리와 같음:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank >= 5 AND year_rank <= 10

  Q: Write a query that shows all top 100 songs from Jan 1, 1985 through Dec 31, 1990
  A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year between 1985 AND 1990

4. IS NULL : allows you to exclude rows with missing data from results

SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE artist IS NULL
** artist = NULL 은 에러남. arithemtic에 NULL VALUE 사용 불가

Q: Write a query that shows all of the rows for which song_name is NULL
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE song_name IS NULL

5. AND
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2012
  AND year_rank <= 10
  AND "group" ILIKE '%feat%'
Q: Write a query that surfaces all rows for top 10 hits for which Ludacris is part of the group
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank <= 10
  AND "group" ILIKE '%ludacris%'

Q: Write a query that surfaces the top-ranked records in 1990, 2000, and 2010
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank = 1
  AND year IN (1990, 2000, 2010)

Q: Write a query that lists all songs from 1960s with "love" in the title
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year BETWEEN 1960 AND 1969
  AND song_name ILIKE '%love%'

6. OR
SELECT *
  FROM tutorial.billboard_top_100_year_end
WHERE year = 2013
   AND ("group" ILIKE '%macklemore%' OR "group" ILIKE '%timberlake%')
**year 2013을 만족하는 것중에서 macklemore, timberlake 들어가는 것들. 2013으로 먼저 걸러냄

Q: Write a query that returns all rows for top 10 songs that featured either Katy PErry or Bon Jovi
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank <= 10
  AND ("group" ILIKE '%katy perry%' OR "group" ILIKE '%bon jovi%')

Q: Write a query that returns all songs with titles that contain the
word "California" in either the 1970s or 1990s
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE song_name LIKE '%California%'
  AND (year BETWEEN 1970 AND 1979 OR year BETWEEN 1990 AND 1999)

Q: Write a query that lists all top-100 recordings that
   feature Dr. Dre before 2001 or after 2009.
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
WHERE "group" ILIKE '%dr. dre%'
   AND (year <= 2000 OR year >= 2010)

7. NOT  : can put before any CONDITIONAL STATEMENT to select rows which that statement is false

SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2013
  AND year_rank NOT BETWEEN 2 AND 3
**This does not make sense, just use comparative operator

SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2013
  AND year_rank NOT > 3
  ##This will give out error. Rather, use:
  AND year_rank <= 3

**NOT commonly used with LIKE & IS NOT NULL
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2013
  AND "group" NOT LIKE '%macklemore%'

Q: Write a query that returns all rows for songs that were on the charts in 2013 and do not contain the letter "a"
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2013
  AND song_name NOT LIKE '%a%'

SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE song_name NOT ILIKE '%a%'
  AND year = 2013

Now done with filtering. Get onto Sorting

ORDER BY

reorder results based on the data in one or more COLUMNS!!!
Default:
- a to z (ascending order) ; 작은것부터 큰거
- ORDER BY 'column_name' DESC : 큰것부터 시작

SELECT *
FROM tutorial.billboard_top_100_year_end
ORDER BY artist DESC

Most recent years come FIRST
Top rank songs comes before lower-ranked songs
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank <=3
  ORDER BY year DESC, year_rank
**year DESC 먼저 sort 하고 그 다음에 year_rank
**예를 들어, 2013년에 year rank 1,2,3 그 다음 행에 2012 년에 rank 1,2,3,....

SELECT *
  FROM tutorial.billboard_top_100_year_end
WHERE year_rank <= 3
ORDER BY year_rank, year DESC
** year rank로 먼저 1순위부터 추리고 거기서 year 가장 최근거 sort
** 예를 들어, 2013, 2012, 2011, ...., 1969 1순위....... 2013, 2012, ..., 1969 3순위 순

Q: Write a query that returns all rows from 2010 ordered by rank, with artists ordered alphabetically for each song.
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year = 2010
  ORDER BY year_rank, artist
-- comments
/*also can leave comment
across different lines */

Q: Write a query that shows all rows for which T-Pain was a group memeber, ordered by rank on the charts, from lowest to highest rank (100 to 1)
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE "group" ILIKE '%t-pain%'
  ORDER BY year_rank DESC

Q: Write a query that returns songs that ranked between 10 and 20 (inclusive) in 1993, 2003, or 2013 order by year and rank and leave a comment on each line of WHERE clause to indicate what the line does
A:
SELECT *
  FROM tutorial.billboard_top_100_year_end
  WHERE year_rank BETWEEN 10 AND 20
  AND year IN (1993, 2003, 2013)
  ORDER BY year, year_rank

0. An Introduction to statistical learning

효효캉 — Tue, 19 Jan 2021 16:56:16 +0900

대학원에서 참고서적으로 읽었던 "An Introduction to Statistical Learning with Applications in R"을 참고하여 머신러닝의 기본과 프로젝트에서 다루었던 모델을 정리하려 한다. 원서는 감사하게도 무료로 온라인 pdf 버전을 다운 받을 수 있다:

statlearning.com/

Introduction to Statistical Learning

statlearning.com

다른 원서에 비하여 설명이 자세하게 잘 되어 있고 머신러닝에 대한 기초 지식이 없는 사람들도 따라오기 쉽기 때문에 추천한다.

1. Basic introduction of statistical learning

효효캉 — Tue, 19 Jan 2021 16:52:29 +0900

Basic of statistical Learning

X_1, X_2, ... , X_p가 주어졌을 때 Y를 구하는 것 (prediction)

X_1, X_2, ... , X_p와 Y의 관계를 이해하고 X가 달라짐에 따라서 어떻게 Y가 달라지는지 이해하는 것 (inference)

- Y = f(X) + ε

X; 독립변수 = 설명변수 = Input variables = predictors, independent variables, features, variables
Y; 종속변수 = 반응변수 = Output variables = response, dependent variable
ε; 오차항 = random error term = independent of X and has mean zero (normally distributed around mean zero)

1. Prediction 예측

앞서 말한 오차항의 평균이 0이기 때문에 y를 predict 하면 다음과 같은 결과가 나온다:

ŷ는 y를 예상한 값이고 다음과 같이 2가지 오차에 따라 그 정확도를 측정할 수 있다:

a) reducible error: 말 그대로 reducible 한 error이다. f^은 f의 값을 예측한 것이기 때문에 오차가 발생할 수 밖에 없다. 이러한 오차는 알맞은 statistical learning technique를 사용함으로써 줄일 수 있다.
b) irreducible error: 줄이지 못하는 오차이다. 앞서 말한 바와 같이 Y = f(X) + ε 이고 여기서 오차항인 ε 는 "independent of X" 하다. 즉, 우리는 ε 를 주어진 X를 가지고 예측할 수 없기 때문에 얼마나 f를 잘 예측하는지 상관없이 ε 에서 나오는 오차는 줄일 수 없다.
- irreducible error가 0보다 큰 이유 (항상 존재하는 이유):
  1. Y를 구하는데 있어서 필요하지만 쓰이지 않은 X가 있을 수 있다.
  2. unmeasurable한 variation을 포함할 수 잇다.
E(Y − Yˆ )^2: average = expected value = squared difference between predicted and actual value of Y
Var(ε): variance associated with error term 오차항의 분산
목표:
- estimate f
- minimize reducible error

2. Inference 추론

- X 와 Y의 관계를 이해하는것

다음과 같은 질문을 할 수 있다:

Which predictors are associated with the response? (identifying the important predictors)
What is the relationship between the response and each predictor?
Can the relationship between Y and each predictor be adequately summarized using a linear equation, or is the relationship more complicated than that?

3. When to use prediction ? vs. When to use inference?

알아내고자 하는 것에 따라서 예측을 쓸 수 있고 추론을 쓸 수 있다.

1. 예측을 쓸 경우:

목표: 마케팅 캠페인 이메일에 긍정적으로 답할 사람들을 찾아내기
Y: 마케팅 캠페인에 대한 반응 (긍정적/부정적)
X: demographic variables

2. 추론을 쓸 경우:

TV, PV/Mobile, Radio 각각의 광고 효과의 데이터가 있다고 치자. 광고의 효과를 측정하는 지표는 sales로 정했을 경우, 다음과 같은 질문들을 통하여 추론이 가능하다:
- 어떤 미디어가 가장 많이 sales에 contribute 했을까?
- 주어진 TV의 광고의 증가에 따른 sales의 변동은 어떻게 될까?

4. Parametric vs. Non-parametric methods

Statistical Learning method은 크게 2가지로 나누어 진다:

1. Parametric

2. Non-parametric

자세한 내용을 다루어 보자.

1. Parametric:

정의: model-based approach이다. 즉, 모델을 정하고 나서 알맞은 파라미터 값을 구하는 것이다.
방법: 2-step
1. f의 functional form/shape에 관한 가정을 한다. 예를 들어, "f is linear in X"라는 가정을 세우면 다음과 같은 식으로 나타낼 수 있다: f(X) = b_0 + b_1 X_1 + b_2 X_2 + ... + b_p X_p
2. training 데이터를 사용하여 모델을 train 한다. 예를 들어, 앞서 언급한 선형회귀에 데이터를 fit하기 위하여 Ordinary least squares를 사용 할 수 있다.
장점: f 전체를 예측하는 것이 아니라 파라미터 셋을 예측하는 문제가 되어서 더욱 간단하다.
단점: 모델에 대한 가정을 하기 때문에 true unknown form of f를 예측하는 것은 힘들다. 이에 대한 대책으로는 flexible한 모델을 ㅆ는 것인데 보통 parameter 수를 늘리고는 한다. 하지만 파라미터 수를 늘려서 더욱 복잡한 모델을 만들게 되면 데이터를 overfitting하는 위험이 있다.

2. Non-parametric methods

정의: f 의 functional form 에 대한 가정을 하지 않고 각 데이터 포인트에 가장 가까운 f 를 추측한다.
장점: 가정을 세우지 않기 때문에 다양한 shape를 시도할 수 있고 따라서 정확도 또한 parametric method보다 높다.
단점: f 자체를 예측하지 않고 적은 수의 파라미터를 예측하는 것이 불가하기 때문에 parametric approach에서 쓰인 파라미터 갯수보다 월등히 많은 파라미터가 필요하다. 따라서 오버피팅의 위험이 있다.

5. Trade-Off between Prediction Accuracy and Model Interpretability

일반적으로 flexibility가 증가할수록 interpretability는 줄어든다.

Inference(추론)을 할 경우에는 flexibility를 높여서 정확도를 높이는 것보다 interpretability에 맞추어 변수들 간의 관계를 알아내는 것이 더 적합하다. 반면,
prediction(예측)을 할 경우에는 정확도가 중요하기 때문에 interpretability는 신경을 쓸 필요가 없다. 하지만 그렇다고 하여 꼭 more flexible한 method이 더 정확한 결과를 도출하지는 않는다. 오버피팅의 문제 때문이다.

[Figure1] 을 보자:

[Figure 1]: Tradeoff between Flexibility and Interpretability

Least squares has relatively low flexibility & relatively high interpretability
Lasso has lowest felxibility & highest interpretability. 그 이유는 lasso는 linear model에 추가적인 fitting procedure를 더하는데 그 결과 계수를 예측하는데 있어서 linear model보다 더 제한적이다. 자세히 말하자면 shrinkage method을 쓰는데 ridge와 다르게 lasso의 경우 중요하지 않은 변수의 계수를 0으로 만들어 준다.
Generalized Additive Models (GAMs) 는 linear model에 특정한 non-linear relationship을 더한 것으로 linear regression보다 flexible하다. 다만, 독립변수와 종속변수의 관계를 선이 아닌 곡선으로 표현하는 것이기 때문에 less interpretable하다.
bagging, boosting, support vector machine with non-linear kernals와 같이 fully non-linear methods는 매우 flexible하지만 interpret하기 어렵다.

6. Supervised vs. Unsupervised Learning

Supervised Learning: 독립변수와 종속변수가 존재함
- Prediction, Inference
- logistic regression, GAM, boosting, SVM, etc
Unsupervised Learning: 독립변수만 존재함
- Clustering

Information Management system of School

효효캉 — Sat, 9 Jan 2021 18:07:50 +0900

학교에 관한 데이터베이스 디자인을 간단하게 구성해 보았다. 다음과 같은 구성으로 데이터베이스 디자인을 해보았다:

1. ER diagram

2. Schema

3. SQL code

4. Normalization

1. ER Diagram

Figure 1: ER Diagram of School

Figure 1 은 다음과 같은 가정을 한다:

- 학생들은 복수 전공이 가능하다

- 휴학을 하고 있는 학생들로 인해 어느 과목에도 수강신청이 되어 있지 않은 학생들이 있다

- Insructors 중에는 대학에서 오로지 연구만 진행하는 경우가 있다

Participation constraints도 존재한다:

- students must have majors and majors must have students
- courses must have students
- courses must have instructors
- instructors should be included in specific department, and vice versa
- departments must provide majors and vice versa
- college must have departments and vice versa
- course must be offered by the department
- major must have core course( s)

Key constraints는 다음과 같다:

- Major can only be included in one department
- department can only belong to one college
- instructors can be working in only one department
- course can only be offered by one department

ISA Hierarchy 는 students, instructors, 그리고 person entity 사이에 hierarchy를 형성하기 위해 사용하였다. Person은 logical sub-components인 students과 instructors로 구성되어 있다. Overlap contraints는 허용되지 않는다. 즉, Person은 동시에 instructor이면서 student일 수 없다. Covering constraints 또한 있다; 모든 사람들은 student 와 instructor 둘중에 하나에는 꼭 포함이 되어 있어야 한다.

2. Conceptual Schema:

Person(ID: integer, FirstName: string, LastName: string)
Student(ID: integer, StudyMode: string, Nationality: string, Age: integer, EnrollmentDate: string) enroll(Grade: string, ID: integer, CID: integer)
Course(CID: integer, Title: string, credit: integer, DID: integer)
instructor(ID: integer, Salary: real, HireDate: string)
Office(officeID: integer, Location: string)
Major(MID: integer, Mname: string, DID: integer)
Department(CoID: integer, DID: integer, Dname: string, Budget: real, StartDate: string) Offer(Semester: string)
College(Coname: string, CoID: integer)
work (ID: integer, DID: integer, Since: string)
HasMajor(ID: integer, MID: integer)
Core(MID: integer, CID: integer)
Offer(CID: string, DID: integer, Semester: string)
Teach(CID: integer, ID: integer)

3. SQL

Figure1 을 참조하여 SQL 코드를 작성하면 다음과 같이 나타낼 수 있다:

4. Normalization

데이터 무결성과 데이터 의존성을 확보하고 중복되는 데이터를 없애기 위하여 각 테이블에 Boyce-Codd Normal Form을 적용하였다. Normalization을 통하여 테이블을 더 효율적으로 사용할 수 있다.

Starting with Person entity, we have R(ID, FirstName, LastName), with FD of ID -> (FirstName, LastName). Since {ID}+ ={ID, FirstName, LastName}, it is in BCNF.

For Student entity, we have R(ID, EnrollmentDate, Age, nationality, StudyMode), with FD of ID -> (EnrollmentDate, Age, nationality, StudyMode). As {ID}+ = {ID, EnrollmentDate, Age, nationality, StudyMode}, it is in BCNF.

For Office entity, we have R(officeID, Location), with FD of OfficeID -> Location. As {OfficeID}+ = {OfficeID, Location}, it is in BCNF.

College entity has R(CoID, CoName), with FD of CoID -> CoName. Since {CoID}+ = {CoName, CoID}, it is in BCNF.

Department entity has R(DID, CoID, DName, Budget, StartDate). Its FD are the following: DID -> (CoID, DName, Budget, StartDate), DName -> DID. {DID}+ ={DID, CoID, DName, Budget, StartDate} and {DName}+ = {DName, DID, CoID, Budget, StartDate}. Therefore it is in BCNF.

Instructor has R(ID, HireDate, Salary, officeID, DID). Its FD are are following: ID-> (HireDate, Salary, officeID, DID) and officeID -> ID. Since {ID}+ = {ID, HireDate, Salary, officeID, DID} and {officeID}+ = {ID, HireDate, Salary, officeID, DID}, it is in BCNF.

Work has R(ID, DID, Since), with FD of ID, DID -> Since and ID -> since. This is not in 2NF. So we decompose R(ID, DID, Since) into R1(ID, DID) and R2(ID, Since). There is no transitive FD so 3NF holds. {ID}+ = {ID, DID}, therefore R1 is in BCNF. {ID}+ = {ID, Since}, therefore R2 is also in BCNF.

For Major, it has R(MID, DID, MName), with FD of MID -> (DID, MName) and MName -> MID. Since {MID}+ = {DID, MName, MID} and {MName}+ = {MName, DID, MID}, it is in BCNF.

For HasMajor, it has R(ID, MID), it has no non-key attributes. Therefore, it does not have functional dependencies.

Course has R(CID, Title, Credit, DID). One course can be divided into different session, therefore can have different CID for the same Title. The FD here will therefore be: CID -> (Title, Credit, DID) and Title -> (credit). By applying Armstrong’s Rules split, we can infer that CID -> Title. We find that the functional dependency is not 3NF give Title -> Credit. Thus, we decompose R(CID, Title, Credit, DID) to R1(Title, Credit) and R2(CID, Title, DID). For R1, as {Title}+ = {Title, Credit}, it is in BCNF. For R2, as {CID}+ ={CID, Title, DID}, it is in BCNF.

Enroll has R(ID, CID, Grade). Its FD are ID -> (CID, Grade) and (ID, CID) -> Grade. As it is not in 2NF, we decompose R(ID, CID, Grade) into R1(ID, Grade) and R2(ID, CID). There are no transitive FD. For R1, {ID}+ = {ID, Grade}, and for R2, {ID}+ = {ID, CID}. Therefore, it is in BCNF.

Core has R(MID, CID). It has no non-key attributes. Therefore, it does not have functional dependencies.

Offer has R(CID, DID, Semester). FD are CID -> DID, (CID, DID) -> Semester and CID -> Sem. As it is not in 2NF, we decompose R(CID, DID, Semester) into R1(CID, Sem) and R2(CID, DID). Closure is {CID}+ = {CID, Sem} for R1, therefore is in BCNF. For R2, {CID}+ = {CID, DID}.

Teach has R(CID, ID). It is a fully-keyed relation. Therefore, it does not have functional dependencies.

Basic security_ 정리노트

효효캉 — Mon, 4 Jan 2021 18:27:42 +0900

Security에는 3가지 종류가 있다:

Theoretic security : always secure under all circumstance (almost impossible except one time pad)
Computational security : practically not feasible to circumvent a service
Based on cost

Information security

def: basis for protecting information assets

Protection measures for info security:

Prevention _encryption
Detection _modification
Reaction/ Recovery

Security goals/ 3 aspects of information protection ; CIA

Confidentiality : prevention of unauthorised disclosure of information 비밀 - We wish to prevent
Integrity : prevention of unauthorised modification of information. 진실성 - We cannot prevent, best is detection, cannot recover really
Availability : prevention of unauthorised withholding 밝히다 of information or resources - We cannot prevent, need to detect, ideally recover

Threat

Security is only desirable when there is a need to protect a system from threat.

Threat - some thing that can possibly go wrong

Attack - threat actually happening

Security threat. : means by which security policy may be breached (loss of integrity/confidentiality)
Countermeasures : controls to project against threats
Vulnerabilities : weaknesses in the system
Attack : realisation of threat (exploiting a vulnerability)

Classification of threat

Deliberate
Accidental

Related to CIA threats:

Exposure of data
Tampering with data
Denial of service

Adversaries 적

Def: people whose aim is to circumvent your security. Intruders also called, but not all adversaries are external to system

종류:

Active

Unauthorised alteration, deletion, transmission, prevention of access to information
Falsification of origin of information

Passive : attempt to get unauthorised access. Don’t touch data, only observe.

Threat, service, mechanism을 구별할 줄 알아야 한다. 예를 들어 threat이 disclosure일 경우 security service는 confidentiality이고 security mechanism은 encryption이다.

Security threat

Possible means by which your security goals may be breached

e.g.) loss of integrity/ confidentiality

Disclosure

Security service

Measure which can be put in place to address a threat

e.g.) provision of confidentiality

Confidentiality -> CIA can be considered as security service

Security mechanism

Means to provide a service

e.g.) encryption, digital signature

Encryption

2 classes:

Specific security mechanisms
Pervasive security mechanisms

Algorithms are used to build mechanisms

Authentication	Entity authentication Origin authentication
Access control	Protection against unauthorised use of resource
Non-repudiation	Non-repudiation of origin: Protects against sender of data denying that data was sent Non-repudiation of delivery: protects against a receiver of data denying that data was received

Empirical Asset Pricing using Machine Learning

효효캉 — Mon, 4 Jan 2021 18:18:15 +0900

1. Empirical Asset Pricing 이란?

- Asset pricing의 경우 CAPM/Fama-French 등 다양한 가격결정 모형을 사용하는 것.

- Empirical Asset pricing은 다양한 데이터를 수집하여 자산 가격을 결정하는 것.

2. Asset Pricing에 Machine Learning을 쓰는 이유

1) Risk premium = Expected return - Risk Free Rate

- 머신러닝은 예측에 매우 전문화되어 있다. 따라서 리스크 프리미엄 측정 문제에 잘 사용될 수 있다.

2) 자산가격을 예측하는 데 사용되는 변수의 종류들은 다양하다. 여러 가지 stock level characteristics 뿐만 아니라 macroeconomic predictors도 포함된다. 변수들이 서로 높은 상관관계를 보일 때도 있는데 머신러닝을 사용할 경우 dimensionality redcution을 통하여 유용한 변수들을 추려낼 수 있다.

3) 기존 방식과 달리 머신러닝은 functional form에 관한 flexibility를 제공한다. 예를 들어 linear model을 쓸 것인지 non-linear 한 모델을 쓸 것인지 여러 모델을 적용할 수 있다. 변수들 사이의 복잡한 관계를 approximate 할 수 있으며 parameter penalizing과 model selection을 통해 오버 피팅과 잘못된 모델 사용을 방지할 수 있다.

3. Data Collection & Preprocessing

다음과 같은 데이터를 사용하였다:

1) Target variable : r_i, t (S&P 500 index의 monthly return)

Yahoo Finance에서 데이터를 수집하였는데 아래와 같은 데이터를 수집하게 된다.

내가 원하는 것은 monthly return이기 때문에 계산해준다:

Market return = Stock return - Risk free rate

Stock return = (end-start)/start

2) Dependent variable

위에 있는 표1에서 r_i, t를 제외한 전부가 다 종속변수로 쓰였다. 각 변수에 대한 설명은 표에 있으니 참조하면 된다.

데이터는 Amit Goyal의 웹사이트에서 수집하였다. 종속변수를 크게 2가지로 나누었다:

1. primary stock characteristics: d/p, e/p, d/e, svar, b/m, ntis

2. interest rate related characteristics: tbl, lty, ltr, tms, dfy, df, infl

3) Data preprocessing

Training Dataset
1938.01 - 1999.12

Testing Dataset
2000.01 - 2019.12

데이터를 testing과 training 셋으로 나눈다. 또한 Date 컬럼을 변수에서 제거했다.

Data Scaling을 진행한다. 머신러닝을 진행하면 데이터를 그대로 받아들이기 때문에 변수들 의 데이터 range가 다른 경우 scaling을 해주는 것이 좋다. sklean 라이브러리에 있는 preprocessing.StandardScaler()를 사용하였다. StandardScaler()는 데이터가 mean 0, standard deviation 1으로 normally distributed 하다고 가정한다.

4. Exploratory Data Analysis

correlation heatmap

Pandas Profiling을 사용하여 상관관계 히트맵 시각화를 진행하였다. correlation r의 절댓값이 0.7보다 큰 경우, 상관관계가 높다고 할 수 있다. 히트맵을 보면 높은 상관관계를 가진 변수들은 다음과 같다:

- lty(long term rate of return) & tbl(treasury bill rate)

- b/m (book to market ratio) & d/p (dividen price ratio)

Target variable인 r_i,t와 나머지 변수들의 상관관계를 보면 significant 한 관계는 보이지 않는다.

5. Results

1) OLS

OLS는 ordinary least squares regression으로 회귀에서 가장 기본적으로 사용되는 모델이다. sum of sqaured residuals를 최소화시키는 방법을 쓴다.

sklearn의 linearmodel.LinearRegression()을 사용하였다.

MSE는 0.00258로 적은 편이다.

여러 가지 변수들 중에 어떤 변수의 featrue importance이 높은지 알아보자.

결과를 보면 feature importance가 0.05보다 높은 변수들은 d/p, e/p, d/e가 있다.

2) Ridge Regression

sklearn.linear_model.RidgeCV를 사용하였다. Ridge regression은 정규화된 linear regression model로 ridge estimator를 모델에 사용함으로써 multicollinearity를 해결할 수 있다. 교차검증 방법으로는 Leave One Out Cross Validation(LOOCV)을 사용하여 튜닝 파라미터를 찾았다. 21.6438을 튜닝 파라미터로 썼고 최종 MSE는 0.002337이 나왔다.

Feature importance를 살펴보면 significant 한 feature는 tbl, lty, ltr, svar, dfr, dfy, tms이다.

3) Lasso

Lasso는 Ridge처럼 linear regression에 페널티를 부과한다. 다른 점이라면 Lasso는 상관이 없는 변수들의 값을 0으로 만들 수 있기 때문에 영향이 적은 변수들을 배제할 수 있게 해 준다.

교차검증에는 5-fold cross validation을 사용하였고 그 결과 튜닝 파라미터의 값은 0.00131로 나왔다. 튜닝 파라미터를 적용하여 분석을 한 결과 MSE는 0.00129로 나왔다.

significant 한 변수들은 다음과 같다: tbl, ltr, svar, dfr

4) Random Forest

sklearn.ensemble.RandomForestRegressor를 사용하여 머신러닝을 진행하였다. 알맞은 파라미터 설정을 위하여 3-fold cross validation으로 n_estimator, max_features, max_depth, min_samples_split, min_samples_leaf, bootstrap 값을 설정하였다.

그 결과 다음과 같은 결과값이 나왔다:

위에의 결과값을 RandomForestRegressor에 입력해준다.

MSE의 결과값은 다음과 같다:

RandomForestRegressor에는 feature_importances_가 있어서 쉽게 feature importance를 구할 수 있다. Significant feature는 tbl, svar, ltr, dfr, dp로 나왔다.

sorted(RF.feature_importances_, reverse=True)

5) Support Vector Regression (SVR)

10-fold cross validation을 통해 튜닝 파마리터를 정했고 그 결과, MSE는 0.002984가 나왔다. Significant features는 ltr, tbl, svar, lty, dfr, dfy, tms, infl, ntis이다.

6) Gradient Boosted Regression Tree (GBRT)

3-fold cross validation으로 파라미터 값을 설정하였고 그 결과 MSE는 0.001864가 나왔다.

다음과 같은 파라미터를 교차검증으로 설정하였다:

결과값은 다음과 같다:

결과값을 입력하여 얻은 MSE:

Feature importance에서 significant한 값은 tbl, svar, ltr, dp, dfr, dfy이다.

7) Long Short Term Memory (LSTM)

LSTM은 RNN의 한 종류로써, RNN의 vanishing error problem을 해결할 수 있다. Vanishing error problem을 간단하게 설명하자면, 과거의 input들의 영향이 시간이 지남에 따라 줄어든다는 것인데, 이러한 RNN의 문제점을 LSTM은 memroy cell을 사용하여 해결한다.

LSTM에 5 hidden layers을 사용하였고 geomatric pyramid rule에 따라서 각 층의 뉴론 개수를 설정하였다. 아키텍쳐의 자세한 구성은 다음과 같다:

각 모델의 MSE를 보면 NN1의 MSE가 가장 작고 흥미롭게도 NN2부터 NN5까지의 MSE가 동일하다.

6. Conclusion

Lasso, GBRT, RF 순으로 예측률이 가장 높았다. 반면, 예측이 가장 낮은 모델은 SVR with linear kernel, LOS, LSTM이었다. 또한 앞서 알아본 feature importance를 종합해보면 가장 자주 언급된 treasury bill rates, stock variance, long term rate of return이 S&P 500 excess return을 예측하는데 있어서 가장 중요한 요소들이라고 할 수 있다.

참고문헌:

Ivo Welch and Amit Goyal (2008). A Comprehensive Look at The Empirical Performance of Equity Premium Prediction.
Shihao Gu & Bryan Kelly & Dacheng Xiu (2018). "Empirical Asset Pricing via Machine Learning, " NBER Working Papers 25398, National Bureau of Economic Research, Inc.