December 7th 2019 is the 78th anniversary of the Imperial Japanese Navy’s surprise Pearl Harbor attack on the capitol ship battle line of the US Pacific Fleet. After that attack there was a round of American elite political and military leaders a collective swearing of “Never Again.” That is, “Never again will the USA be so surprised by a foreign enemy.”
Yet despite that, America has indeed been “surprised” in exactly the way of Pearl Harbor repeatedly since 1941. The Korean war is one example five years after WW2 ended. The Soviet Invasions of both Czechoslovakia and Afghanistan in 1968 and 1979 are two others It was certainly an intelligence surprise on 9/11/2001 with the attacks on the World Trade Center in NY City and the Pentagon in Washington D.C., and the “surprise” of there being few/no Weapons of Mass destruction in post 2003 Iraq, and Iran’s recent drone and cruise missile attack on Saudi Arabian oil refining facilities.
The reason for this pattern of failure boils down to the forgotten and unlearned — frankly impossible for American elites to learn — intelligence lessons of Pearl Harbor. Those unlearned lessons being that the interlocking patron-client political relations inside the American federal civil government, military and intelligence organizations lead to narrow self-interested group think over the concerns of outside reality. And that this tendency towards self-interested group think is at its absolute worse when facing a foreign enemy with a police state internal security system that is running a campaign of strategic deception and denial.
If that “worst case” foreign enemy sounds a lot like Imperial Japan, the People’s Republic of North Korea, China, the Soviet Union, Iraq and Iran. It means you have paid attention to both American history since Pearl Harbor and to current events.
The Role of Clientelism in American Intelligence Since 1941
Clientelism is the one-word description of patron-client relationships whose synonyms include words like patronage, cronyism and corruption.  For the operation s of the modern American state, the terms “Military-Industrial Complex” or “Iron Triangle” have grown up since Pearl Harbor to describe the aligned interests of elected civil, military, intelligence and other federal bureaucratic elites in the Federal Government who are patrons of large industrial clients of Federal government largess.
It wasn’t until Samuel Huntington’s book The Soldier and the State – the Theory and Politics of Civil-Military Relations in 1957 that “Civil-Military Relations” describing these relationships in academic terms was written.  And it wasn’t until Amy Zegart’s 2000 book Flawed by Design: The Evolution of the CIA, JCS, and NSC that how these various elites operated in creating the intelligence community that simply can’t ever do it’s post-Pearl Harbor job of making sure of “Never Again.” 
The following is from Zegaet’s book marketing on Amazon:
“…Zegart asks what forces shaped the initial design of the Central Intelligence Agency, the Joint Chiefs of Staff, and the National Security Council in ways that meant they were handicapped from birth.
Ironically, she finds that much of the blame can be ascribed to cherished features of American democracy―frequent elections, the separation of powers, majority rule, political compromise―all of which constrain presidential power and give Congress little incentive to create an effective foreign policy system. At the same time, bureaucrats in rival departments had the expertise, the staying power, and the incentives to sabotage the creation of effective competitors, and this is exactly what they did.
Historical evidence suggests that most political players did not consider broad national concerns when they forged the CIA, JCS, and NSC in the late 1940s. Although President Truman aimed to establish a functional foreign policy system, he was stymied by self-interested bureaucrats, legislators, and military leaders. The NSC was established by accident, as a byproduct of political compromise; Navy opposition crippled the JCS from the outset; and the CIA emerged without the statutory authority to fulfill its assigned role thanks to the Navy, War, State, and Justice departments, which fought to protect their own intelligence apparatus.
Not surprisingly, the new security agencies performed poorly as they struggled to overcome their crippled evolution. Only the NSC overcame its initial handicaps as several presidents exploited loopholes in the National Security Act of 1947 to reinvent the NSC staff. The JCS, by contrast, remained mired in its ineffective design for nearly forty years―i.e., throughout the Cold War―and the CIA’s pivotal analysis branch has never recovered from its origins. In sum, the author paints an astonishing picture: the agencies Americans count on most to protect them from enemies abroad are, by design, largely incapable of doing so.”
Zegaet’s description was achingly prophetic come 9/11/2001. Yet her conclusion’s regard the National Security Council moving beyond those “Flawed at the Beginning” limits had a very short sell-by date in the post-Iraq invasion Weapons of Mass Destruction debacle. The NSC had no more escaped the Clientelism trap of the JCS and CIA than did the Departments of Navy, State and War in December 1941.
That trap also has another name, “Group Think.”
Clientelism, American Intelligence, and the Role of Group Think
The amazing thing about the failures of American intelligence of the interwar-period of mid-1930’s to 1941 on Japan and that of Saddam Hussein’s Iraq from 1991 to 2003 are the intelligence group think parallels in both that over rode available data in the face of consistent denial and deception efforts by the foreign regimes involved in both wars.
The dominant historiography of American intelligence failures before Pearl Harbor blames the racism of American (and British) military intelligence officers, Flag Rank military officers and elected political leaders for missing the arrival of Imperial Japan as a military “peer competitor” (to use a modern term of art) in the late 1930’s to 1941…until the the reality of torpedoes of the Kido Butai arrived in the hulls of the Pacific Fleet’s battle line. This historiography’s apogee was reached with John Dower’s 1986 book War without Mercy: Race and Power in the Pacific War. 
I’ve always cast a gimlet eye on this “Racism was the sole cause” historiography as being an easy and far too simplistic approach that does a disservice to both history and the the people involved. Yes racism played a role. But to say it was the _Sole Reason_ for Intelligence failure denies the Imperial Japanese agency. (Which is another form of racism, when you think about it.)
The competence of the Imperial Japanese military and state had a whole lot to do with their success at Pearl Harbor, and everywhere else, until the Battle of Midway and the Guadalcanal campaign. In short: The enemy gets a vote. That’s why they are called “the enemy.”
So, if Racism was neither the sole nor the primary cause of American intelligence failures of that time. You have to ask the question “What was the cause of this intelligence failure?” Or perhaps more appropriately, “What did American intelligence know about the Japanese, when did it know it, and why did it get so much wrong by Dec 6th 1941?”
There is a historiography doing just that. In chronological order see the following articles of this emerging historiography:
- Ralph Lee Defalco III “Blind to the Sun: U.S. Intelligence Failures Before the War with Japan” (2003),
- R.J. Hanyok’s “Blinded by the Rising Sun: Japanese Radio Deception Before Pearl Harbor” (2006), “Catching the Fox Unaware”—Japanese Radio Denial and Deception and the Attack on Pearl Harbor” (2008) and “How the Japanese Did It” (2009),
- Bob Bergin’s “Claire Lee Chennault and the Problem of Intelligence in China,” Studies in Intelligence (2010), and
- Justin Pike’s “”Blinded by the Rising Sun? American Intelligence Assessments of Japanese Air Power, 1920-41” serialized in three parts on the Balloon’s to Drones web site in August and September 2017 
Defalco’s and Bergin’s works cover the sole American who got it right about Imperial Japanese air power pre-WW2 — including the capabilities of the A6M Mitsubishi Zero fighter — then retired captain and future USAAF General Claire Chennault. General Chennault’s memoir “Way of a Fighter” makes clear he was as much a racist towards the Japanese as any other American military officer of his times…but he was also right about their aerial war making capabilities. When everyone else in American pre-war intelligence was wrong. Defalco and Bergin explain why that was.
Hanyok’s works are on how Imperial Japanese naval intelligence determined what Anglo-American signals intelligence capabilities were. How the IJN planned the denial and deception measures to blind them as to the movements of their Kido Butai carrier fleet and how well they executed that plan up to Pearl harbor. Taken together, they paint a picture of Imperial Japan as a fell “high tech” foe, an enemy fully versed in the latest in electronic intelligence…and the means to deceive it.
Pike’s serialized work does a forensic analysis of classified American intelligence on the Japanese from the end of World War I to the beginning of World War II. Pike finds little if any outright racism. What he does find is that American intelligence was highly accurate in the 1920’s to early 1930’s, when the Imperial Japanese allowed open access to their society. This is how he closed part one of his series:
Taken as a whole, American intelligence assessments of Japanese air power during the 1920s were highly accurate from the strategic and industrial spheres down to the tactical and technological level. Lax information security measures within Japan provided American observers with a remarkable level of freedom. American reports assessed the ability of Japan to fight a protracted war in the air, where aircraft production figures, industrial efficiency, innovative aircraft design, strong pilot training programs, and sizeable pilot reserves were critical to achieving success. The correct conclusion was Japan did not yet possess air power that could seriously threaten the Western powers, but this would begin to change in the 1930s.
When the Imperial Japanese Military closed that access in the early-1930’s due to the war in China. The American intelligence work increasingly diverged from the changing Japanese reality and started filling the lack of intelligence with regurgitated open source articles that repeated the 1920’s tropes of lack of originality in design and backwardness in tactics and equipment.
In time, the weight of old and incorrect “conventional wisdom” meant saying saying anything else became threat to an intelligence officer’s opportunities for advancement. Thus, by 1937, when the Imperial Japanese were making truly original and innovative aircraft a generation past anything they previously copied. And when Claire Chennault started providing the naval attache’ at the US embassy with accurate reports and captured Japanese aircraft components from downed planes. American military intelligence officers simply could not go there. Bucking the “conventional wisdom” — group think — was too professionally dangerous given the decade and a half of ingrained and by then horrid intelligence reports that had become belief system of the flag rank patrons above them.
This is how Bob Bergin put it regards Chennault’s bumping conventional wisdom group think in his “Claire Lee Chennault and the Problem of Intelligence in China” –
Intelligence was now a major concern. Within the US military establishment, “current intelligence on the Orient just didn’t exist,” he wrote. He looked for ways to learn about his enemy, and what he learned he shared with the US embassy. From Japanese airplanes that crashed during the first air battles he salvaged equipment and sent the best of the materiel to the US naval attaché. With the Japanese advancing on Nanking, the attaché secured it in the safest place he knew, aboard the US gunboat Panay. Two days later the Panay was attacked by the Japanese and sent to the bottom of the Yangtze. With it went Chennault’s collection of Japanese military equipment.5
Chennault continued to collect everything he could about the Japanese Air Force, but his efforts made little impression back in Washington. In 1939, the Chinese captured an intact Japanese Type 97 “Nate” fighter. Chennault had it flown in extensive tests against comparable British, American and Russian aircraft and compiled a thick dossier on the Nate’s construction and performance. He believed it was one of the best acrobatic airplanes ever built—“climbs like a skyrocket and maneuvers like a squirrel”— and turned the dossier over to US military intelligence.
In time Chennault received a letter from the War Department. It said that “aeronautical experts believed it was impossible to build an airplane with such performance… with the specifications submitted.” In late 1940, he visited Washington and brought with him data on the first model “Zero.” That information was never disseminated. “American pilots got their first information on its performance from the Zero’s 20-mm cannon a year later over Oahu and the Philippines.”6
Such was the level of group think in American military intelligence about Japan’s aerial military power at that point. That had the Japanese aerial equipment Chennault shipped on the USS Panay actually made it to the USA for analysis. The 1940 American military intelligence report would have played a game of “…the Japanese copied from fill in the blank foreign plane here” quoting the same “aeronautical experts” who had dissed the 1939 Chinese report on their captured Japanese Type 97 “Nate” fighter.
From Chennault’s Type 97 Fighter to Saddam’s Real WMD’s.
The inability of 1930’s American military intelligence to digest valid intelligence data against the conventional wisdom of elites are all too familiar to students of the 2003 Iraqi WMD debacle. Iraq’s Saddam Hussein regime was a Soviet style police state and as such was past master of the sort of deception and denial the likes of which has bedeviled American intelligence since World War II. In particular, it had utterly surprised American intelligence in the aftermath of the 1991 Gulf War when it revealed in April 1991 a almost completely indigenous nuclear weapons program that used “obsolete” Electro-Magnetic Isotope Separation (EMIS) AKA “Calutrons” and had and 10.97 kilograms of 80% enriched (bomb grade) U-235. 
This Iraqi 10-year Deception and Denial success — from the 1981 Israeli bombing of Iraq’s French built Osirak nuclear reactor at Tuwaitha to the 1991 invasion — utterly biased future American political assessments of intelligence community product. On issues of Iraqi WMD’s, the intelligence community had lost it’s professional credibility with the executive branch filling political elites of both US political parties.
While most Democratic party and foreign policy elites now blame the Pres. George W. Bush Administration for lying in front of the UN and Congress in the run up to the 2003 conquest of Iraq. This Jeremy R. Hammond’s 2012 Foreign Policy Journal piece titled ‘The Lies that Led to the Iraq War and the Persistent Myth of ‘Intelligence Failure’ is typical of this “conventional wisdom.”  There is a systematic exclusion of the October 2001 Hart office building and postal Anthrax attack and how it affected the thinking of the Pres. George W. Bush administration in this assessment.
Whatever the Mueller FBI had said about it’s two USAMRIID suspects in the Senate Anthrax case.  Before the 2003 invasion of Iraq, the CIA had credibly connected the anthrax silica transport media used in the Senate Hart Office building and US post offices to a report of a then unidentified Iraqi silica compound impregnated by mustard gas against Iranian forces.
See this passage from the 2009 H.W. Beuttel article, “Chemical Weapons and Iranian Casualties in the Iran-Iraq War: A Further Note and Update”  —
Further, as regards the lethality of mustard gas in particular, deaths per wounded soldier in WWI were about 2%. If 16,000 were indeed killed by mustard, then this would suggest Iran’s chemical wounded from mustard alone were on the order of 800,000 or eight times the highest total Iranian acknowledged chemical casualties! However, if you are going to die from any chemical agent, mustard is a good bet. Of 1,221 hospital deaths from chemical agents experienced by the AEF in WWI, 600 (50%) were due to mustard. By contrast, the arsenic-based German “Blue Cross” (diphenyl chloroarsine) produced only 3 deaths in the AEF out of 580 total casualties from this agent (0.5% lethality)!106 There are other reports of as many as 5,000 Iranian chemical deaths from mustard gas, and the vast majority of post war chemical injured are mustard casualties. According to the CIA, Iraqi forces used an unidentified silica compound impregnated by mustard gas against Iranian forces. This substance was delivered in White Phosphorus shells. The silica compound reduced the amount of mustard gas the shell could carry, but actually decreased the dose rate required to produce a casualty, resulting in effectiveness five times the standard shell. It apparently helped the agent create a vapor rather than a contact hazard among those exposed. It was noted that Iranian soldiers exposed to mustard gas had unusually high amounts of respiratory injuries as opposed to the more common skin blistering.107 The higher proportion of lung injuries among Iranian soldiers would increase the agent’s overall lethality.
107 “Mustard Gas Used By Iraq in War with Iran,” cia_62648_61898_01.txt
When the above, secret in 2003, CIA report was added to the following 9/11/2001 open source reports:
1. The disputed report that Czech intelligence reported a meeting between an Iraqi agent and the 9/11 ringleader Mohammed Atta that might have been the Iraqis handing off anthrax spores.
2. The fact that one of the 9/11 hijackers was treated by a doctor in Florida for an infection prior to the attack.
3. Which afterwards, subsequent to the anthrax outbreaks, the doctor who had treated the hijacker looked up his medical books and thought that the infection was cutaneous anthrax.
You had a “Fog of War” first impression impression that utterly convinced the George W. Bush Administration it was facing an irrational regime with biological weapons of mass destruction…one that had used them
There was simply no way most American military or civilian intelligence officials in their right minds were going to get in the way of that political patron belief and say “There are no WMD in Iraq.” And the few voices that did said so were ignored, side lined or removed because of the Iraqi nuclear bomb shock of April 1991.
Jackson Landers Sept 2016 Smithsonian.com article “The Anthrax Letters That Terrorized a Nation Are Now Decontaminated and on Public View” captures some of the George W. Bush Administration push against the intelligence community, but not the security classification hidden, wide eyed, terror engendered by the CIA’s report of Iraqi silica-mustard gas delivery media. 
Deception, Denial, Iraqi Fictitious WMD’s & the Perils of “Never Again”
It is at this point in the story where American political-intelligence elite group think and clientelism meets the Deception and Denial games of Saddam Hussein yet again. While all the above was going on with American elites after the 9/11/2001 shock. The survival needs of the Saddam regime after the 1991 Gulf War required that the Iraqi Army and Republican Guards believe that Saddam still had nerve gas filled multiple rocket launchers to put down revolts. So after Saddam had surrendered his stocks of WMD to the UN following the 1991 Gulf War. He started a deception and denial campaign aimed at his own population that he still had WMD’s.
It was successful. 
So successful that high level Iraqi defectors in the late 1990’s and early 2000’s brought that “Saddam still has secret WMD caches” belief to Arab. Israeli and Western intelligence agencies.
Thus you again have in the 2003 Iraq invasion the triple witching hour of American intelligence group-think and institutional clientelism meeting foreign enemy deception and denial campaigns as we had with the Imperial Japanese in December 1941.
In a real sense the lessons of the surprise of Pearl Harbor and the 2003 Iraqi WMD debacle were buried because they were too painful to learn.
Namely, no bureaucratic intelligence structure or political reform will fix the clientelism or group think inherent to the political-military intelligence structure of the American state that existed in 1941, in 2003 or now. America and American elites in particular were not hurt badly enough to be willing to change. It will take another Pearl Harbor, with mass casualties directly to those group-thinking American political-military-intelligence elites, before their replacements will learn.
It hasn’t happened yet…but it will.
Notes & Sources:
From Wikipedia, the free encyclopedia
 Huntington, Samuel P. The Soldier and the State – the Theory and Politics of Civil-Military Relations. Cambridge, Massachusetts: The Belknap Press of Harvard University Press, (1957). ISBN-13: 978-0674817364 ISBN-10: 0674817362
 Amy Zegart, Flawed by Design: The Evolution of the CIA, JCS, and NSC 1st Edition, Stanford University Press; 1 edition (August 25, 2000) ISBN-13: 978-0804741316, ISBN-10: 080474131X
 John Dower , War without Mercy: Race and Power in the Pacific War, Pantheon; (1986) ISBN-13: 978-0394751726, ISBN-10: 0394751728
 Full article citations and links listed below:
Ralph Lee Defalco III (2003) Blind to the Sun: U.S. Intelligence Failures Before the War with Japan, International Journal of Intelligence and CounterIntelligence, 16:1, 95-107, DOI: 10.1080/713830379 | Published online: 15 Dec 2010
R.J. Hanyok “Blinded by the Rising Sun: Japanese Radio Deception Before Pearl Harbor” World War II Magazine, December 2006
Robert J. Hanyok (2008) ““Catching the Fox Unaware”—Japanese Radio Denial and Deception and the Attack on Pearl Harbor,” Naval War College Review: Vol. 61 : No. 4 , Article 10.
Robert J. Hanyok “How the Japanese Did It” December 2009 Naval History Magazine Volume 23, Number 6
Bob Bergin, “Claire Lee Chennault and the Problem of Intelligence in China,” Studies in Intelligence, Vol. 54 No. 3 (June 2010) Pages 1 – 40. https://archive.org/details/DTIC_ADA523664
Justin Pyke “Blinded by the Rising Sun? American Intelligence Assessments of Japanese Air Power, 1920-41: Part 1 – The 1920s”
August 24, 2017
Justin Pyke “Blinded by the Rising Sun? American Intelligence Assessments of Japanese Air Power, 1920-41: Part 2 – 1930-1937”
August 29, 2017
Justin Pyke “Blinded by the Rising Sun? American Intelligence Assessments of Japanese Air Power, 1920-41: Part 3 – 1937-41”
September 6, 2017
 Claire Lee Chennault (Author), Robert Hotz (Editor), Richard Edes Harrison (Illustrator) Way of a Fighter: The Memoirs of Claire Lee Chennault (History United States Series) 1st Edition, G. P. Putnam’s Sons; 1st edition (1949), ISBN-10: 0781248132 ISBN-13: 978-0781248136
 Iraqi Nuclear Weapons
 Jeremy R. Hammond, The Lies that Led to the Iraq War and the Persistent Myth of ‘Intelligence Failure’
Sep 8, 2012
 US News & World Report and later Washington Examiner Senior Political Analyst Michael Barone followed the twists and turns of the FBI Anthrax investigation closely for 10-years until the FBI cases against USMRIID doctor’s Hatfield and Ivins both collapsed in 2010 without any further suspects. See the followings (USN&WR links require the archive.org Wayback Machine):
Who’s behind the anthrax scare?
Web exclusive 10/17/01
September 25, 2006
Anthrax and al Qaeda
November 13, 2007
Who was behind the September 2001 anthrax attacks?
01/01/10 6:59 PM EST
Anthrax attacks still unexplained
01/29/10 4:05 PM EST
 H.W. Beuttel, “Chemical Weapons and Iranian Casualties in the Iran-Iraq War: A Further Note
and Update,” pages 21 – 32 The International TNDM Newsletter Volume 3, Number 1 Summer 2009
 Jackson Landers “The Anthrax Letters That Terrorized a Nation Are Now Decontaminated and on Public View”
SEPTEMBER 12, 2016
 If you have enough time and a strong stomach. You can tease out the out story of Saddam’s ultimately self-destructive WMD deception and denial campaign past the…asset covering…in this report.
REPORT TO THE PRESIDENT, MARCH 31, 2005
SUMMARY OF CONTENTS