as we move forward in space will be looking at how we integrate those capabilities into the other tool kit that belongs to our war fighters. >> absolutely. well, we could go on. we have some great additional questions, but i'm getting the high sign. please join me in thanking joe votel, a national treasure, for a wonderful conversation. >> thank you. i was going to try and get you fired up here because the initial point of recession, we're a little bit of subdued. so, i want you to feel free to get enthusiastic here. we've got a great panel coming. so, lots of cheering for them, and we're going to have a great discussion. so, welcome back. before we jump into this next session, let me remind everyone

that for the plenaries here and the maryland ball room, you can email questions to the panel. and hopefully we'll have a logo up that says questions @intelsummit.org. lair is going to get your questions. by the miracle of technology, they're going to appear on an ipad with vince here. vince, if you didn't get the word, at some point larry's going to tell you last question, okay? so, it's now my pleasure to introduce the moderator of our defense intelligence plenary lieutenant general vincent stewart, u.s. marine corps -- come on, come on -- retired. >> that's the part i was waiting

for. >> and vince is the founder and ceo of stewart global solutions, an international consulting firm. he retired from the u.s. marine corps in april of this year after more than 38 years of service to the nation. so, join me in thanking vince for his service. a career intelligence officer, vince served as the 20th director of the intelligence agency where he was the first african-american, the first jamaican american, and the first marine to hold the position of director dia. general stewart earned degrees in national strategic studies

from the naval war college and war strategies from the defense university. please join me in welcoming former director, good friend of industry and my friend, vince stewart to the stage. over to you, vince. >> all right. i don't usually let folks waffle on about my career like this, but since this is chuck's last time, i let him get away with that a little bit. thank you all for joining us this afternoon. we've got a distinguished panel here this afternoon. if you don't know them, you're in the wrong place. i'm not going to ask them to do an introduction, tell you who they are and where they're from. you've got deputy secretary under the armed forces, jeff kruse, and jake. i want to go through a quick

round of questions. i want you to have your questions ready for follow up. what i'll do with the question ts is give them a chance to set the stage, talk about their highest priorities, what their concerns are, how the industry could help. and then maybe on the back end we'll ask the question i used to get all the time "what keeps you awake at night." so, we'll do a table-setting kind of sessions and we'll follow up -- and we'll do this kind of quick pace, fast break basketball. so, if a question is answered and we don't like the answer, we want to follow up, we'll jump right in. so, we're going to start with ms. bingam. lots of conversation about machine learning and the implications for the intelligence community. you've drived project maven as the pathfinder for machine learning. can you give the audience a sense of where we are with project maven, jake, some of the challenges you've seen, and some

of the ways we can be helpful as partners? >> sure, vince. thank you. it's great to be here. love this event every year. i'm really happy to be here participated. so, maven. i always love the chance to talk about maven and artificial intelligence machine learning. i like to preface it by saying we're at the infancy, the machine learning phase. we would like to get to ai but we have a ways to go. maven is a pathfinder. when we were here about two years ago, we were on track for within six months of authority to proceed to get an initial minimum viable product capability out to the field. since that time, they have scaled pretty significantly in terms of increasing algorithm performance, scaling geographic locations. my boss was just out in theater a couple weeks ago and got to see it in action. scaling across different isr platforms, we're in dozens of legacy platforms now. and then scaling across different missionaries, not just

full motion video but other area of intel. so, we're proud of that but have a tremendous amount of work to go. just a couple of points to make on what we're learning because i think the key about being a pathfinder is you learn and apply that to broader activity that is the jake in particular is undertaking. one, i would say get your users involved early. this is -- for us, it's been all about fielding, get that minimum viable product which this is agile development and practice. get that out to the field, work with the users, and build from there, scale it from there. second thing i would say is where we've learned the most has been in what we call the ai pipeline. so, writing the algorithms probably the easiest part. it's the up-front data access, data labelling which is incredibly manual, integrating at the back end in the weapons systems, doing the accreditation of all these different dod system ls. so, those are the big areas where we're learning.

now as we go into this year, we've got almost two years under our belt now, it's out there in enough capability that we need to now see change. we need to see work force changes. we need to see efficiencies. we need to see cultural changes as a result of bringing ai into the field. that's what we're looking for in the next year. >> okay. thanks. susan, much has been said about the building of the analyst of the future. while the analytic trade craft will largely remain the same, the analyst of the future will still be inundated by a vast amount, growing amount of data. with all that data that the analysts will have to swim their way through, what's dia doing to think about big data analytics, and where does mars, a program we all hear about, fit into this analytic framework? >> so, there is no question that analyst of the future is something that dia is trying to

hone in on and define more specifically. clearly mission number one is providing the analytic assessments to all of our customer base, timely, accurate, insightful, comprehensive understanding of the information out there, providing insight, and ideally understanding the operational environment. but you're right, the data that's available to the analyst to craft those assessments only increases by the minute. not only that, but the operating environments that we have to assess are evolving. space is a great example of that. so, the types of data and the exquisiteness of that data are increasingly challenge for our analysts. you named mars as certainly top of the list, if you will, of the initiatives that dia is undertaking to try to help our analysts make sense of all that information and fundamentally deliver more insights. so, mars is step one. for those in the room who may not be that familiar, it's our effort to really develop and expand the data environment in which our analysts -- it

accesses many more databases in one fail swoop, if you will, allows for additional databases we don't currently have in an automated fashion, and hopefully we'll leverage ai technologies as they mature to allow more rapid and creative insights on the part of the analyst. mars is not it. we're doing much more in the realm of open source intelligence. open source has been around for quite some time. what dia is doing new is structuring our effort against open source. even just earlier this year, general ashley made some decisions to establish more efficiently a career path if you will for open source cloak tors and dia. we're establishing an open source integration center, putting efrpts around pockets in the enterprise. the intent there is to bring increased discipline around the activities that many of our analysts and collectors are undertaking now to drive some maturity into that process and ultimately to allow the analyst

to leverage open source in a much more official trade craft compliant and useful way. so, that's number two that i would highlight. and then finally i would highlight jox. none of this happening -- none of this is effective without a strong jox foundation. jox has been around for 25 years and has to continue to evolve to meet the customer demands which are only increasing even outside the ic user abase. the types of things people are using jox for is modernizing. we're undertaking investments and approaches to do exactly that. the time is now given the need to support mars, to support o-sent and over all futures. >> i'm probably going to come back to jwix later.

i'm going to skip jeff for just a second because admiral whitworth and i talked about the challenges of produces intelligence in a timely manner in this environment. and one of the things as susan was talking about, i was thinking about when was the last time we used the national intelligence estimate? that's a pretty static document. it's almost obsolete at the point of publication. can you talk a little bit about the challenges of generating real time intelligence in an estimated sort of environment, in a spiral sort of -- talk about some of the challenges of developing intelligence in this crazy world, randall. >> yes, sir. thank you for that question. the chairman and the joint staff have made it clear we need to focus on global integration. we're not going to look at just regional. we're going to look at opportunity as it applies to other adversaries, taking advantage.

that creates so many permutations of potential estimates. so, to your point, when we draw, let's say, a national intelligence estimate or something that just stays on the shelf as the 2019 edition, when you're trying to get that integrated and that global, presents a problem. how do you go back and fill in variables that contributed to that particular estimate with some real time urgency and speed? enter what we were classifying as our giant concept for intelligence operations to. your point, it's a spiral estimate. it's basically trying to get going with multiple variables so we can rest assured that at least at some juncture all of the data is being looked at. we're being queued to possibly reassess for a new estimate. >> so, in case our industry partners didn't pick up on that, static assessments, the static

summaries that drive decision making doesn't fit in the modern very dynamic world. how does industry help us to build that very dynamic spiral estimate as the world changes, as the variables changes, how do we see those different permutations so that that drives decision making at the policy and strategic level? so, that's an area that just having almost every publication that we put out there as intel estimates begins to be obsolete at time of printing. how do we make that more dynamic? that's the requirement. i'm going to leverage jeff a little bit here. i've heard lots of questions today and comments about great power competition. jeff spent three years as a j 2. he got to see one of the great powers carry out its strategy. i'm going to ask jeff to just

kind of help us characterize the challenges you saw on the front end of the pacific command specifically as china walked through its strategy. so, i'll give you an extra 30 seconds so you can -- >> it's an easy one for any good intel officer. let me add my thanks to everyone joining us today. great conversations and thank you sir. i think i would preface my conversation with china and the same pay com turned into pay come. in 2014 and 2015 i happened to be running for general in yukon and working through problems russia was going through, conducted a handful of operations they wanted to do under a certain amount of cover. and the question for us is how do you illuminate that, and how do you kwens the rest of your allies and partners or your own

department of defense the threat that in this case russia presented? the second piece is way back when, if you go back ten years when i was a student at the national art college, we had a piece of china, two column by two rows and it was china continuing to rise and china unable to maintain that momentum, and china being friendly to western powers or china being an adversary. a decade ago, nobody was picking the quadrant of china being able to sustain its rise internationally, economically, and militarily while still presenting a challenge to the international order. 2016, show up in pay com which is probably where you wanted me to start this story, and the first piece that i really came to grips world war all the assessment that assessments that we made a decade previously, china had far surpassed those estimations. we used to rely on a handful of

things about china, that we had qualitative and quantitative and experiential ability to do whatever they needed to do and go beyond. we were far more advanced in the all of the areas that mattered. it turns out china had spent five, maybe six five-year plans methodically over the years systematically addressing the areas where we had an advantage over the years that they could observe. so, the first piece, my first probably 12 to 18 months was illuminating and really understanding where china is in that journey that they were on and what kind of threat it presented. so, great partnerships with the national community. and it is remarkable what china was able to do in that period of time. the period at the tail end of that first 12 to 18 months was the crafting of the new national defense strategy. so, that really paralleled back

to how do we convince the rest of the department of what china looks like and what are their goals and objectives going forward? and the piece that i would offer to you is that at the macro level, i consider china an open book. they're academic right, they plan as a nation, and they move out in a very large way in a public fashion. we know their national objectives are to restore or achieve regional, to displace the u.s. as a global power, and to change some of the international organizations out there to be more advantageous to china's authoritarian model. so, if we know that, it becomes then what we worked on for the second 18 months which is the hard part, what do you do about it? so, the piece i would offer to the crowd here is that while we used to rely on china being able to build a lot of stuff but not

being able to train with it, not being joint enough, they have worked through all of those issues and they are presenting as we have seen in the most recent exercise which concluded not that long ago a very dynamic capability to employ high end equipment against a scenario that should cause a variety of regional partners to take note of what they're attempting to do. i would say china has tried to keep their intentions below the radar scope so it doesn't elicit the kind of response that we would normally take. and they were certainly counting on an ability to use ties in the academic business and other communities to make it challenging for the united states to make some decisions about what we're going to do to address that. so, where we sit today in my view and the challenge for all of us is what is it that we can do today. we're in the competition space today.

china would certainly like us to take a little to no action, to focus on the urgent versus the long-term issue. and i think what i would offer is the important piece for us is to understand great power competition and do well in that conflict today, posture ourselves so, you know, two, four, five, twenty years from now we don't find ourselves in a position we don't want. just tee it up that way. >> i would love to jump on that. we do have a new secretary, deputy secretary. the good thing is here they've maintained our department-wide focus on the national defense strategies, so we're not seeing any changes there. prioritizing china and russia and modernization. i think jeff hit it on the head is we are in this strategic competition phase right now. and, you know, the department, we do order battle well, we do force well, we buy aircraft and tanks and ships well. it's that gray zone space that

we don't do as well. it's looking at the nexxus between economic and national security. i think about china's one belt one road initiative. military planners really care about who's operating what ports well. care about who's operating railways. we care about resilient communications, telecon networks. so, all of these things are happening now. and if we can't effectively be present and counter in that competitive and frankly economic space, you're setting yourselves up for not being as successful if and when the balloon goes up. just another thing i mention there is just technology theft as well as -- jeff hit on this too. we're seeing them in a pretty concerted effort go after specific technologies. in the past we've been fortunate

to be able to maintain that technology and the military advantage. when you're stealing it now, we're going to see the same technology that we're doing r&d on now, we'll face it five or ten years from now on the battlefield. so, that technology advantage is eroding. so, we're doing a lot more on the security front to address that. >> is there something industry can do to help us in securing that technology and managing the supply chain so that china doesn't take advantage of the great r&d that we do? what are the things we would like industry to do? i'll put that up as a jump ball. >> i'm sorry. so, the supply chain insights transparency management is increasingly a concern for us. obviously we've been focused on it particularly in the it space for years. what we're finding is that's not enough. that's too narrow of a viewpoint. we've tried to certainly open our aperture and work more closely with industry part fer hads to understand the entirety

of the supply chain coming through the dia doors in our case and understanding the tangents that each of the supply chains go into. it's daunting. there's no question about it. but we have to do it. and we're increasingly looking to our vendors to come to us with transparent display, if you will, of your supply chain so that we can understand it, so that ecowith have confidence in it and really make sure that everything that comes through the front door is uncompromised throughout, whether it's equipment. it could be cameras, chairs, furniture, i.t. equipment, i could go on. we have to think more seriously and work together on the supply chain because understanding it takes multiple views and multiple phases of both supply chain and on the government side too. >> it's knowing that you're a target. there are multiple methods that the chinese are using right now to go after our technology and intellectual property. so, cyber means, clear defense

contractors. we know you are a target and we know it's your unclassified information, your subs that are the target. basic cyber hygiene, encrypt data at rest, don't connect thumb drives to mission systems and then the internet. you would be surprised some of those that do that. but basic cyber hygiene. knowing having strong insider threat programs, better controlling your sensitive but unclassified material. so, there's a whole host of things that you can do. and we are in the department treating security as a missionary and no longer an a admin. >> for the first 25 year of my clear i had the luxury of not paying a lot to economic intelligence. for a large part we relied on large partners or niche partners to do that for us. the last three years i've spent

more time on economic intelligence and understanding the connections between economics and security issues and industry's ability to work through those issues is welcome. to echo a little bit on the cyber hygiene, defending and protecting your networks, but also having the same approach to counterintelligence that we are doing. we are scaling up our approach to counterintelligence due to what is being relaid to as china's approach and other peoples' approach to direct access vectors. the direct access vector to what they want is not necessarily the path they're going to take. china's global presence gives them a lot of opportunities. for me it is asking industry to be cognizant of your networks, people, connections, vendors all across the globe. and the last piece is our national strategy really depends on allies and partners. anything industry can do to help protect and secure the networks

of friends, partners, and allies because of our desire to connect to them at speed and scale with data means we're going to have to rely on the security of their networks. and there's a application to u.s. national security. >> so, let me echo industry partners a little bit and get the reaction. you guys make it so hard for us to bring you the best technology that solves your problems today. you talk about acquisition reform, talk about acquisition agility. you built all kinds of structure, universal needs statement, joint universal, all sorts of things that fix the symptoms but doesn't allow industry to bring the best ideas, the best capability, the best technology to you today. where's the department? where are we in really allows industry to help us be good partners? anybody want to jump on that grenade? because that's what i hear from industry. you make it really, really hard

to help. the first person that told me that was jake jakobi when i took over at dia. he said i've got stuff i can't share with you. where's the department on acquisition reform, acquisition agility? what can we do better? >> if i could phone a friend and have ellen up here, they're doing a lot there. but i can't -- i -- >> sorry. >> there's a lot of truth to that. >> i told them no got you questions. that may have been a got you question. >> no, i think there's a lot of truth to that. me working with joe kernen who has not spent time at the edge of the spear but also spent time at industry, we know there's plenty of talent and ideas out there. we're seeing the services whether it's castle run or other activities or vehicles to bring in ideas. with maven, we brought in over

40 different industry partners from the bigs to some of the smaller dot com start ups. i'll say we need to do a better job practicing what we preach. but what we say agile d development, there's a ba out there for something give me something in six months, give me something in six months. don't give me a proposal for how to do it. we put six months out there as a bogey and we sprinted and had capabilities across the spectrum of bigs to smalls actually deliver that we are able to integrate. don't bring us proposal. bring us capability and we'll build it if pr there. >> i was going to go with minimal viability product. you could talk about what that means. i think you just did. don't bring me proposal. don't bring me 18 power pooint slides. bring me something that's usable today. anybody else want to jump on that? >> i can speak to the

congressional side of this. even in the context of mars, what we're finding is still in some cases a level of discomfort on the hill with all things agile development because it's not well-defined by definition. particularly as they're looking for us to justify resources and investment, they're saying when is it going to be delivered? what are you going to be able to use it? when am i going to see something concrete? how much is it going to cost? we're saying not sure because we're going through agile development, and testing things out, and we're going to like some and not like some and have to adjust. some are having a hard time embracing that and supporting that. yes, the department has work to do for sure. and we all have work to do on leveraging the flexibilities that already exist, but we also have to work with congress on the receiving end, if you will, because they're fundamentally critical of course to us proceeding. >> trey, did you want to -- oh, you didn't. >> i'm very happy with the spots. >> you have one of the biggest

challenges in terms of warning in the j2, trying to get insights. and i remember many times the u.s. forces career commander wanted 72 hours of unambiguous warning. have you solved the warning problem yet? and if you haven't, what do we need? what are the challenges, and how do we get after solving the warning problem for you? >> thanks, sir. there's a lot being written about the warning problem. i think a lot of people are citing we have a warning problem. at the same time the question is do we have a warning problem or an only nishs problem? do we have a listening problem or a data problem? this kind of gets to a natural segue. what keeps you awake at night? is emeril jakobi here? i don't know if you remember 25 years ago, you said i just want it you wiall. i want it all.

i want all the data. and that was -- at that time, it seemed like a huge quest. but it was at least a goal we could set. now i'm more worried about what we're not going to at least even assess, what we're -- what hits the cutting room floor. so, that really keeps me awake. on the warning problem, what hits the cutting room floor. so, we need an application. we need a series of applications. we need some additional help so that we have the certainty that at least data has gone somewhere and has been processed through -- i'll just offer perhaps machine learning, ai, to at least queue us -- what i was talking about before. new variables, you need to know this. you need to at least consider this. that's where i would say -- that's kind of answer one on what keeps us awake. and it is directly tied to warning. the second would be also related, and that is assessing

ourselves. how do we know that we are assessing ourselves? this is tied into the joint concept for intelligence operations. this is kind of the second part which is a spiral estimate of our own readiness. so, as we get into this globally integrated world and so many things could go wrong, so many warning problems could be out there, opportunity costs, et cetera. how do we know that we have all systems on go? how do we know that the federation we set up is working? how do we know we've got the right number of analysts allocated to different warning problems that might be related? so, other ait's a complex answe to what seems to people to be a very easy question. >> jeff. >> can i had one thing to that? in my vignette for warning that was instructed mofor me, we had usks. all the warning problems that

are written with precision and tracked by the most number of people to provide the most collaborative results, that is the warning problem. but, you know, quite frankly, there's three major aspects to it. one of them is warning of just internal stability for north korea. so, when we just swapped out -- and i say "just." it's been a little bit. usfk commanders came through the head quarters in a series of office calls. my first conversation was this: sir, what do you need warning of? he rattled off three things which required a major rewrite of the warning problem. to me that's always step one is understanding what do you want warning of and then designed the warning problem and being comfortable if two weeks later what i want warning of changes. >> i don't have any questions et why, so either we're having a really great conversation and you guys don't have any conversations or we put you to sleep. but that's okay.

we can't -- we've got to talk a little bit about cyberspace operations. u.s. cyber command is coming up on nearly ten years now from the early days. how are we doing in terms of intel support to cyber stations? we thought our way through the right policies? are we meeting the requirements from the joint staff? how does dia organize for support to cyberspace? or is it just another intel thing that's plugged into the multi-domain world? jump ball. >> i'll take a stab. having seen it at least from nsa's perspective, i think we're postured better than people think mainly because cyber is a domain that is inherently intelligence driven. people are in the domain.

intelligence analysts are in the domain when they do the collect and help an operation to occur. to think that it's broken and that there is not sufficient intel support to cyber perhaps reflects some people who aren't entirely familiar with what actually happens inside of an ops room. that might seem a bit harsh, but it's true. it's very hard to describe just how much intelligence is going into a cyber operation without it being inside the ops room. so, i'm actually more optimistic than most people on this issue. >> okay. anybody else? susan, you want to talk about jwix a little bit? what's it look like and what do you need to transform? what does the future look like? >> i wish i had an answer to that. what we're trying to do is gauge from the broader community exactly what they think they need from it. it continues to evolve.

as i mentioned earlier, it started as a core ic system. it was exactly that for many years. what we've seen over the last five years plus is really an expansion of interest in jwix in the system. there's multiple reasons for that. it's fundamentally trust and the reliability of the system and the security of the system. it's understanding that frankly that's all the information you could need would reside on that system instead of going back and forth between super net and jwix which is efficient to many users. we're trying to generate more of that conversation of if we were to create jwix today what would it look like, the ic, dod, and broader. we're looking at this is an opportunity to have exactly that conversation because there is such a demand signal. so, we're trying to understand that so that as we head down our path of modernization, we're

addressing all of those concerns and those potential scenarios for where jwix will go. so, more to come on that conversation. >> no longer an intel collaboration environment. it is a command and control. it's a decision support. it's an intel environment. it is all of the above to all customers at all times. and the real question is is that the right model? and if it's the right model, what does it mean for us designing the next version of jwix 4.0 or whatever the heck it is. it's certainly evolved from here's a intel platform at the highest level to every combatant command, j3, j5, azure jwix box. what's the partner environment that allows us to collaborate in that space? there are lots of different versions out there, but something we really to think our way through. all right.

jeff, you've been -- did you want to -- jeff, you've been in jake now for how long? >> been where? >> where are you now? >> i work for mr. purnen at usdi. not the jake. >> i thought you were going ointo the jake. okay, maybe -- oops. maybe i shouldn't announce that. >> that's another air force general. >> maybe i should get to know the air force. one of the great challenges we have today is awful lot of data being collected on all of us. and what do we do to protect our data internal to your organizations, data that's collected on us? are we doing anything to make sure that that environment is clean for our intelligence

professionals? and what can industry help us to do to protect our data if anything? jump ball. >> i guess, you know, there's at one level there's the -- we clearly have laws, policies, regulations. there's a lot of dod directive and instruction on how we protect information. but then there's the actual mechanics of it and the technology of -- heck, we've got the background investigation mission coming to dod in less than a month now. and as we look at rebuilding that i.t. capability, you know, it's ensuring that it is all of our, everyone that has a clearance or credential, that your information is protected. so, it's building all of those cyber security elements in up front into the system. so, i think there's a technology piece to it as well that we'll continue to need help on.

>> here's the thing that i want industry partners to think about. all of our data's available out there to any of our adversaries. and even the data that we try to protect is available to our adversaries. we have closed circuit tvs and cameras in every city that's capturing biometric data. what does that mean for us not only as a society but for intelligence professionals? how do we protect that day? how do we make sure your adversaries aren't able to use that data? so, how industries are able to think through biometrics, closed circuit tv, all the day that that's out there on the dark web that exposes all of us, but more importantly our sensitive collectors? how can industry help us to solve that problem. that's where i want to go with that question. >> you think about the super squirrely stuff we do, our human intelligence, counterintelligence. the last 15 plus years of ct has

been focused on vetting taliban and whatnot in afghanistan. but when you have a sophisticated russia or china, those skills have atrophied over the last decade and a half. so, we're working on rebuilding it. you're rebuilding in this modern technology environment where there's constant surveillance, smart cities. everything out there on the internet that our agencies are having to think about that, human ci, cover, all those aspects that are core intelligence missions. >> all right. dodc commercial satellites as alternative to national technical needs. >> i'll say as a compliment to national technical needs you're still going to have your need out there for some pretty exquisite high resolution capabilities and other features that those satellites have, you

know, maybe some additional protective measures given where the threat has gone in space that you are willing to pay a premium on. at the same point in time, god, is it an exciting time to be in commercial space right now. i visited one place a couple of months ago where they're fine tuning their production line to pump out six, seven satellites a day. it's unfathomable for someone six or seven years. there are really exciting things there. we absolutely from nro to others have to take advantage of that. the other piece of it where industry can help us quite a bit is not just on the sensing side but on the tped side. you go into a ground station today, it is manual, labor intensive. you've got people moving around bars on computer screens. when you get into the constellations of the size we're talking about, you can't have a person in the loop doing that manually. so, all the automating tasking,

all the collection optimization, the processing side of it, moving algorithms and processing even upstream further, freeing up your analyst to do other things, there is a whole manual labor intensive linear set of processes out there that we must do better. and if we're going to be dynamic as we were talking about earlier and much more responsive to the threat environment we're seeing vis-a-vis china/russia. >> yes, go ahead? >> i would offer as a former j2 for u.s., a place we have partners who would really appreciate that commercial imagery has a perfect application there as well. it's for partnering as we were talking about. it provides them at least with an avenue. >> i could add one or two vignettes to that as former j2. the commercial imagery we've

seen over the last few years have been crucial and helps get the most out of national architecture. for example, indo pay com was a lot of open water. how do you track open ocean areas? how do you track north korean ship to ship transfers of elicit oil? a lot of times that was only a certain period of density with national resolution and being able to stitch those together cohesively in a way that gave me the best picture i had and could provide my boss was always a combination of the various pieces together. so, i think that's really the future. how do we stitch that together and then back to the point on the ability to extract information out of commercial imagery and making that readily available. trey and i just happened to be having a conversation earlier today about synthesis. there's a lot of data. how do we take all of that down

to those two or three key take aways that we need to make decisions in a high-speed priority environment? and the ability to extract data out of commercial imagery is an important business line to continue to pursue. >> so, this one came from the audience, but i was wondering about this earlier. i'll talk about machine learning, talk about artificial intelligence, talk about the technologies coming into the work force. is our work force ready for this? what are we doing to make sure work force culturally ready for this influx of technology? or is it the same work force? it might be the same work force, they'll adapt over time. any thoughts on what are we doing to prepare the work force? >> from dias perfective, it's a little bit of both. work force adapt already, are hungry for it, are eager to continue to evolve their own

capabilities and their habits frankly on how they leverage technology. but that said, there's no question that some of it's going to have to be trained. some of it's going to have to be hired to some extent. and what we're trying to do is really hone in on what does that look like? what skills do we have to teach? what skills do we have to seek out and recruit? and defining that is still part of the puzzle, if you will, that we're trying to piece together on it. but the work force in many respects, we have such a great work force that they're very adaptable. they like experimenting. they're innately curious. you put a new piece of technology in front of them and they will play with it and figure out what it can do for them. they'll take full advantage of it. it's a question of how do we leverage that and encourage and that and develop that where rounding it out where we have the gaps with the hiring and training. >> i would agree with that. we have analysts obviously who are sourced by dia.

we are dia. and especially younger crowd are looking for advantages all the time. they're always looking for an advantage. so, a new application issing g to be welcome. i don't think we're going to run into a situation where analysts are going to be in competition with machine learning environment. they're going to find it complementary. they're not going to find it a subsubstitute. they're going to find it an addendum, something that queues them tie better product. >> how do we train them? the training that we do today is probably not the same sort of training and preparation we need for the future analysts, for the future work force in this environment? is there something we would like industry to think through in term of how we train to be successful in the next era? i want you all thinking about virtual training. i want you all thinking about

immersive increasingly complex training. i want to think about how painful it would be to have an instructor standing on the podium lecturing as we prepare our work force. what's the different training model? is there a different training model? >> i can envision a situation where we would put emphasis on certification to use the tool so that the analyst knows the left/right limits of the product and of the queuing coming out of the tool and we probably need to ensure that they're witting to knowing the tool is probably off at this juncture. this is based on what could be a bad variable. i need to regroup and try again. we went through this actually a long time ago with maven and everything that's happening. we actually looked at the automation of point dropping and the targeting community. and there are some tools that made things very quick, but they

also cut a few corners in the interest of speed. so, as we thought about it, we said we want this tool, but at the same time, we want to know where it potentially is dangerous to precision and to accuracy. and that was the basis of certification for using the tool. tool is actually used. it's not a substitute for the good ole' fashioned way of dropping points. it just happens to complement the process. i could envision analytically a very similar environment. just know what your left/right limits are in terms of tolerances so you can queue to yourself and to your chain of command that this was based on machine learning. we might need to take another round. >> it also goes back to just we need to preserve the hard critical thinking skills of our analysts, especially with all these different data sources that are even more readily available at their fingertip. whether it be the machine learning tools, traditional imagery, or signal intelligence.

even i think publicly available information, we are using that so much more, but you have to be able to discern how reliable, how credible is this, and what kind of decisions are being made off of it. there's an evolution of the trade craft and those critical thinking skills that have to evolve with the technology and all the different data we're getting our hands on. >> lots of the services are talking about multi-domain operations. you know, when i started this business, the army and the air force did their thing and the navy and marine corps did their thing. then we got to joint out to grenada and talking about land battle. now we're talking about multi-domain. what does that mean for the intelligence community? for the j2 and the joint staff, you have operations in a globally integrated world.

what are the challenges of the intel support to multidomain operations in a globally integrated environment? >> sir, i think i can go back to how to assess the opportunity risk as we're beginning just a bit too fixated on a regional problem. that is not an inherit part of our train craft. let's just imagine we have training -- we have decision tools for something as simple as moving from point a to point b in our smartphones, we use either google maps, waze, something like that to help us assess how quickly, what's the best way to move from point a to point b. you're going to use different choke points. just image they are different variables. everybody's done it in here. you just use testing the monte carlo to make sure it's correct. now imagine you have an armada of cars trying to do an objective as well. but there are others who are

having armada of cars and they actually want to confound your solution. imagine all the permutations of that. that starts to get very complex when you're thinking about the monte carlos for opportunity costs and how an adversary might take advantage of your dilemma in a particular place. how does an analyst efficiently succinctly address that? that is not something that we necessarily train to. >> i would add there's a data side on the back end of this. multidomain would require one service to sense, another service to shoot, and perhaps a third service sink a ship. how do we all have that same picture that we can have common understanding and that requires interoperability and data standard that we have yet to achieve but continue to work on? and then there is the data

integrity of the data that flows across there to get after trey's point, how do you ensure that nobody's putting false data in there so if you're doing long range of horizon fires between a shooter who can't see the target and the center that you're not causing fires to go astray or on your own folks. there's a tech side of multidomain operations i think that underpins it. once we work through that piece, then it becomes the ttps and the trust and confidence in how that would work in a global integrated environment. >> i don't want to speak for admiral coaler and the navy because i think these things are still going on. but let me talk as a naval officer. i think the jury's still out as to if we're talking about the domain of space, talking about the domain of cyberspace and how specialized this particular -- these domains are, we have to ask ourselves whether generalization as we're rearing especially officers is the right path or whether we might need to

think very seriously about specialization. i was having this conversation with admiral jakobi not long ago. right now we tend to seek generalists and might need to adjust that in cyberspace going back to your specialized craft. might need to keep people at keyboard for a long time. as opposed to back and forth between some of the domains we have within our services. i think the air force is already moving in this direction. >> suzanne, you were going to comment? >> so, mars, the vision more mars, is to enable multi-domain operations as well. fundamentally one of the key efforts under mars is to understand and try to pull together in some form or fashion various capabilities of the individual services we're going to be using, leveraging, how they bring them together with common data and common access to that data and feeding it back and forth in and out of mars as needed during operation, too. that's a fundamental precept of mars looking forward is how exactly to enable that feature.

>> yeah, there's a good amount of plumbing we still need to do in our community. i'll use dcgs as an example where, yes, you can share -- it was supposed to be the common ground system and we've seen each service has developed its own kind of bespoke capability, so although you can share products the raw data isn't necessarily interoperable. interoperability, data standards, formats and architectures talked about earlier. there's a trust piece, you have to trust that someone else is going to provide data, you can do something with it or act on somebody else's data. the missile defense community, they've had to work through that in the last ten years or so. i think we need to apply some of that learning to other areas as we think through multi-domain. >> talk a little bit about ocent.

creating an open-source center. some thoughts about how that might work. implications of if you do this, you're going to bump into u.s. persons data. we thought our way through the policy. just spend some time expanding on open source center. >> sure. come october is when we kind of open the doors, so to speak, on a more formal center that we've had ever in the past and it's going to grow over time and evolve over time so part is a structural change for us, organizing the activities more specifically to include dedicating people into respective centers or commands, who really will be kind of the hub of the ocent activity in that location, but then as you indicate, part of it is what is the trade craft behind it? what are the rules behind it? again, i think because so many of our analysts and pocketers have been kind of sort of doing it on their own for many years, you know, the imperative is

clear we have to bring some discipline to this particularly when it comes to u.s. persons to make sure everybody is doing it appropriately and accurately and oversight perspective that we can stand by it. it's a significant change for a number of reasons for us. but it's going to be a, you know, an evolving effort because we're not going to wait until we have all the answers figured out before we embark on this. we're going to jump right in and kind of fashion it as with go, but it's been years in the making. the good news is we've finally figured out this is fundamentally something that is, in fact, structural, there is a dedicated career for it for many of our folks, there's enough of a career there and enough of a discipline there that we can call it as such and manage it as such. so i think what we're trying to figure out, though, too, is what

does the enterprise do? we're focused first and foremost on dia. clearly the enterprise as a whole, whether it's the services, the broader commands, what have you, they're going to have a role as wiell and that's going to be kind of the next step we're going to have to define is how do we leverage the enterprise when it comes to ocent, figure out who really can do what, who brings a comparative -- to the problem set and knit it together. it's going to be a continued effort even after we open the center. >> could i add to that? i think one of the grate applications of machine learning and a.i. is in this realm, not necessarily finding medium in these large volumes of data, it's doing at butribution of th data, can we use a.i. machine learning to help discern is the person behind that particular bot or eapost or whatever tied u.s. persons. i think it's great opportunity there for a. ii., machine

learning. >> i agree. can't say enough about ocent as actually form of an early warning especially in theaters where we just don't have sufficient collection. the speed is unmatched. and so you really need that vetting behind that particular one liner to find out if this is an early warning we need to pursue or early warning we need to ignore. >> all right. in the intelligence community's assessment, what extent will china dominate 5g communications, how prepared is the u.s. government to protect both government and commercial 5g networks from foreign threats? can you jump on that one? >> so if i can take it, i don't know if i can speak for the broader intelligence community, i would say they -- it's one of their national priorities. they have put a lot of effort and resource behind wanting to dominate in the 5g arena. i'd say technologically, i don't think that they're there yet so it's not a foregone conclusion.

we will continue to have to stay after it. but this is an area where we in the department are spending a good amount of time on and working with the intelligence community as well. first off, just to make sure that folks understand the security risks of it, whether it's our -- and our foreign partners, whether it's our european our middle east, other asian partners, is there is a clear -- it may be a great package deal today, financing tied up with a bow, but you need to look long term at the security risks and the onus i think is on us in the department to also articulate what the military and security implications are of relying on a chinese or huawei-built 5g telecommunications architecture. for starters, you have a china national intelligence law that mandates that -- that companies and individuals, you know, compel them to cooperate with the chinese government, but on the military impact side, whether it's intelligence sharing or mobility, there are

tremendous amount of military areas that would be affected when we do not have trusted telecommunications capabilities with our close partners. and it's been a challenge working this with our allies and partners for them to see beyond some of the near-term economic benefits but to look at those longer-term security risks and getting the intel and defense, national security folks, then talk to the commerce business community and hop those stovepipes. we're working in the department with a research and engineering group so dr. griffin and dr. porter are leading an effort to create an integration test bed so whether you're u.s. or other companies, i would note, there are other companies out there that are probably leading beyond the -- the u.s. does not have an end/end solution. they're working on developing an integrated test bed so that companies can come and test out

and evolve their technology and work the integration pieces of it so there are viable alternatives to a chinese 5g system. >> okay. we'll start on your end, trey. i think you touched on this a little bit already. what keeps you up at night? anything else? i'm looking for the black swans. looking for the things we go, it's russia, china, iran, north korea, oops, it was -- >> yes, zblir whsir. >> what keeps you -- >> i'm going be honest with you, very few enemies or adversaries keep me awake. it's more the knowledge of ourselves and readiness of our own systems ensuring that we're not missing something. missing the big story, putting the pieces together, the things we've already talked about today. that's what keeps me awake. >> yeah? >> i would absolutely say the same thing. there's no adds vversary or potential adversary that keeps me up at night. it is about our ability to sustain over time a long-term disciplined approach for the various problem sets that are

out there. how do we address the urgent needs and the important needs concurrently and can we maintain that? so that's probably what keeps me up at night. your black swan question is a little bit different in my mind. for me, that is about an area where there's potential for escalation that we have not been through that escalation cycle previously and know how to handle. so india/pakistan border, india/china border, they've worked through those issues methodically over time. there's historical precedent in some of thaez arose areas but t continue to change and those are the ones that i would say are worth watching to ensure we understand where we're going and are providing best advice, assistance, to friends, allies, around the globe to work through those issues. >> yeah? >> so i think for me, it's the risk that we assume every time we have to make a tradeoff decision. so clearly, you know, we have

many more demands than we can ever meet by way of requirements. and our resources just won't keep up. regardless. so we are almost on a dale lay b basis having to make choices. obviously, there's risk in every choice that we make and we con communicate that risk all we want, but just worry that something's going to result as a result of our decisions and that risk is either not going to be appreciated or we're not going to be able to respond. >> we have a tremendous community out there across defense and national intelligence, folks that are working 24/7, 365 days a year, so i don't know that there's anything specifically that keeps me up at night. i actually would echo what trey was saying. i remember -- he and i just thinking over the last couple of days, he and i were on a phone call last friday and we were asking ourselves, hey, have we -- do we have the policy in place so that the folks that need to take action, you know,

whether over the weekend or on a holiday, you know, that they have the authorities, they have the policy, they have the guidance, to do what they need to do. and it's making sure we've thought through those and that we, you know, i'm in policy up at osd, making sure we provided that frame and guidance to them so they can go off and do what they do best. >> okay. i'm coming down the line now. give you one last time to talk to industry. what are your top one, two, three, priorities, what do you think in this you could do to help? start with kari. >> helping us modernize our very manual labor-intensive lib ynea processes. the more i see, the more i uncover in this community, there are a lot of them we fine tune and are bespoke that we need help on, so that i'd say, helping us evolve in areas like a.i., maven, mars, getting those scaled, getting them out to the

field, and then i would be remiss if i also didn't talk the security pieces. you know, with the china and a russia who are intent on getting after our technology and our intellectual property and our people, we have to protect that. we have to make sure that we have a trusted workforce and we'll need industry's help in all those areas. >> so, i'll emphasize pie chain one more time, but then, obviously, with mars, and mars is just the best example to date we have of working with industry and asking industry to challenge our assumptions on what we think we need to help us really think through and think a little more creativity about what we need. and that obviously takes some understanding and constant engagement to live in our shoes a little bit so you can, in fact, do that. i do worry ae all kind of fall into that trap ofand asking y you to deliver it.

we have to flip that and you have to challenge us on that a little bit more. >> data interoperability and standar standards, propriety data standards. we're past that and moving past that. the business model of the future is about integrating and understanding data. and the more we partner with industry, and then collaborative development. if we ever do solve the agile acquisition kinds of things, it means partnering with industry from the very beginning of what are we trying to do, how do we do it, how do we fail together, then how do we deliver some subset of what we started? there's a business model somewhere in there, industry does it well. there's a path for the government in there and partnering with us to help design that and making congress comfortable with that sort of strategy. >> we're the functional managers for three disciplines. warning, collection management, and targeting. we haven't talked too much about targeting, but i'm going to offer three things in that

order. in warning, just helping us not sacrifice accuracy and thoroughness, completeness, for speed. and collection management, we need to help -- we need some help automating the process, automation for collection management is dated. then in targeting, we always need help. this will be the case from now until eternity, just helping distinguish enemy from non-enemy. it's one of the hardest things we do in the business. >> okay. we're going to stop there. i ask you to give the panel a round of applause for spending some time with us. [ applause ] hopefully we didn't get too many jump calls and thank you, all, very much, and -- weeknights this week, we're featuring american history tv programs as a preview of what's available every weekend on c-span3. tonight, garry adelman of the american battlefield trust covers the whole civil war in 56 minutes. this talk kicks off a night of programs from a gettysburg

heritage center symposium. you can see it tonight starting at 8:00 eastern here on c-span3 and enjoy american history tv this week and every weekend on c-span3. in his first public remarks since leaving the trump administration, former national security adviser john bolton talked about north korea. he said military force needs to be on the table. >> one, is the possibility, limited though it may be, of regime change in north korea. second, we should look at and discuss with china, and we should have done it long ago, aiming toward the reunification of the peninsula under a freely elected government like that in south korea. and third, if you believe, and you may not, that it is unacceptable for north korea to have nuclear weapons, at some point, military force has to be an option. now, this is, obviously, the

most controversial subject and many people say it's just unimaginable. unimaginable that you would use military force. so let me quote to you the words of general joe dunford, chairman of the joint chiefs of staff, on his last day, i might say, as chairman. he's done an outstanding job. he said this to the aspen institute seminar in the summer of 2018. on this question of what's unimaginable. general dunford said, "but as i've told my counterparts, both friend and foe, it is not unimaginable to have military options to respond to north kor korea's nuclear capability. what is unimaginable to me is allowing the capability to allow nuclear weapons to land in denver, colorado. my job will be to develop

military options to make sure that doesn't happen." i think general dunford was completely correct. >> you can see john bolton's entire speech followed by q&a at the center for strategic and international studies in washington, d.c.. we'll have it for you tonight starting at 8:00 eastern on c-span. the u.s. intelligence community is looking into how a space force could change intelligence gathering. a discussion now on emerging threats to national security and what that will mean for the intelligence agency's space policy. the armed forces communications and electronics association and the intelligence and national security alliance are hosts of this summit. >> i'm tisch long, i'm chairman of the board of insa. i'm very honored to be moderating this very distinguished panel here this morning. and i think we're going to have some fun. at least that's what i told them backstage. okay. on the pan