Jaynes Notes (WIP)

Table of Contents

• Chapters 1-2
• Chapter 3

Chapters 1-2

These three things.

1. Probabilities represented as real numbers.
   
2. Common sense applied to these numbers.
   
3. Consistency: Equivalent Knowledge -> Equivalent Conclusion
   Allow you to derive the product and sum rules. (Cox’s Theorem)

P(AB|C) = P(A|C) * P(B|AC)
P(A+B|C) = P(A|C) + P(B|C) - P(AB|C)

Any truth defined in terms of atomic propositions can be represented in DNF form as a disjunction (logical sum) of conjunctions (logical products) of atomic propositions.

The derived rules allow you to reduce any such DNF to a computable form.

Exercise 2.3

SHOW 0<=P(AB|C)<=P(A|C)
0<=P(B|AC)<=1
0<=P(A|C)P(B|AC)<=P(A|C)
0<=P(AB|C)<=P(A|C)

SHOW P(B|C)<=P(A+B|C)<=1
P(B|C)<=P(A+B|C)<=1
P(B|C)<=P(A!B|C)+P(B|C)<=1
0<=P(A!B|C)<=1-P(B|C)  
0<=P(A|C)-P(AB|C)<=1-P(B|C)
P(B|C)<=P(A|C) + P(B|C) - P(AB|C)<=1 
P(B|C)<=P(A|C) + P(B|C) - P(AB|C)<=1 
P(B|C)<=P(A+B|C)<=1

SHOW IF P(A|C)+P(B|C)>1 THE BOUNDS ON 0<=P(AB|C)<=P(A|C) TIGHTEN
0<=P(X)<=1
P(AB|C)<=1
P(A|C)+P(B|C)-P(AB|C)<=1
P(A|C)+P(B|C)<=1+P(AB|C)
P(A|C)+P(B|C)-1<=P(AB|C)

SHOW IF P(A|C)+P(B|C)<1 THE BOUNDS ON P(B|C)<=P(A+B|C)<=1 TIGHTEN
0<=P(AB|C)
0<=P(A|C)+P(B|C)-P(A|C)-P(B|C)+P(AB|C)
0<=P(A|C)+P(B|C)-P(A+B|C)
P(A+B|C)<=P(A|C)+P(B|C)

Chapter 3

This chapter is about sampling theory, where a lack of understanding has been “retarding the progress of scientific inference”. You can derive sampling theory (and almost all of probability theory in fact) straight from the product and sum rules.

# "MULTIPLICITY OF R" (BINOMIAL COEFFICIENT) 
import math
n = 4
r = 3
n_factorial = math.factorial(n)
r_factorial = math.factorial(r)
n_less_r_factorial = math.factorial(n-r)
print('-----------------------------')
print(f"{n} choose {r}")
print('-----------------------------')
print(f"n={n}, r={r}")
print('-----------------------------')
print(f"n!/r!(n-r)!")
print(f"{n_factorial}/{r_factorial}*({n}-{r})!")
print(f"{n_factorial}/{r_factorial}*{n_less_r_factorial}")
print(f"{n_factorial}/{r_factorial*n_less_r_factorial}")
print(f"{int(n_factorial/(r_factorial*n_less_r_factorial))}")
print('-----------------------------')

Exercise 3.1
n! gives permutations for whom the sets of r and non-r values are equivalent in all but order. It’s fun to think of these things like a slot machine.

The nonobvious (but kinda obvious) symmetries:

1. For any Bournoilli Urn type problem, if the only knowledge you have is the starting conditions of the Urn, the probability of any draw is M/N.
   
2. Information’s utility for inference is symmetric with respect to time.

The method of calculation:

1. Resolve X into mutually exclusive (X1,X2,…,X3). Doing this in a way such that the mutually exclusive propositions are easily calculable is a skill.
2. Apply Sum & Product

Invariant under permutation:
Is jargon for “Order doesn’t matter.”

Exchangeable distribution:
The probability for a sequence of events is independent of their order.

Expectation:
Weighted average of possible values according to their probabilities.

When you are reasoning about expected future events, you can take the inverse of the event not occuring, instead of trying to add potentially overlapping sequences in which the event does occur.

A probability is numerically equal to the expectation of a fraction.

Exercise on PG67

(using product rule)
WE WANT P(R_k|R_later,B)
USING THE PRODUCT RULE
P(AB|C) = P(A|BC)P(B|C) 
P(AB|C) = P(B|AC)P(A|C)
P(A|BC)P(B|C) = P(B|AC)P(A|C)
P(A|BC) = P(B|AC)P(A|C)/P(B|C)
SO WE CAN SAY
P(R_k|R_later,B) = P(R_later|R_k,B)P(R_k|B)/P(R_later|B)
BY SYMMETRY
P(R_k|B) = M/N
BY COUNTING 
P(R_later|B) = 1 - (COMB(N-M,s)/COMB(N,s))
P(R_later|R_k,B) = 1 - (COMB(N-M,s)/COMB(N-1,s))
PLUGGING IN THESE VALUES WE GET
P(R_k|R_later,B) = ( [1 - COMB(N-M,s)/COMB(N-1,s)] * (M/N) ) / ( 1 - COMB(N-M,s)/COMB(N,s) )
WHICH SIMPLIFIES TO 3.56

(using 3.72)
1, 2, 3, 4, 5, ..., k, , ..., n, ...
s: k+1 -> n
T: reds in s
M: red ball count (through n)
W_later: All s draws are white
B: No replacement. N. M.
FROM 3.72
P(R_k|R_later,B) = E[(M-t)/(N-s)|T>=1,B)]
P(R_k|T=t,B) = (M-t)/(N-s)  
P(R_k|R_later,B) = 1/(N-s) * (M-E[T|T>=1,B])
SOLVE FOR E[T|T>=1,B]
E[T|T>=1,B] = E[T|B]/P(T>=1|B) = s(M/N)/((1-(COMB(N-M,s)/COMB(N,s)))
If you plug in this value to the prior equation it can be transformed algebraicially to 3.56.