[Tutor] stx, etx (\x02, \x03)

[richard kappler]

I have a file with several lines. I need to prepend each line with \x02 and
append each line with \x03 for reading into Splunk. I can get the \x02 at
the beginning of each line, no problem, but can't get the \x03 to go on the
end of the line. Instead it goes to the beginning of the next line.

I have tried:

#!/usr/bin/env python

with open('input/test.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = ('\x02' + line + '\x03')
f2.write(s)

as well as the same script but using .join, to no avail. What am I missing?

regards, Richard

[Mark Lawrence]

Post by richard kappler
I have a file with several lines. I need to prepend each line with \x02 and
append each line with \x03 for reading into Splunk. I can get the \x02 at
the beginning of each line, no problem, but can't get the \x03 to go on the
end of the line. Instead it goes to the beginning of the next line.
#!/usr/bin/env python
s = ('\x02' + line + '\x03')
f2.write(s)
as well as the same script but using .join, to no avail. What am I missing?
regards, Richard

Your line has EOL on the end. Getting rid of the unneeded brackets try:-

s = '\x02' + line[:-1] + '\x03\n'
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

Thanks for the reply Mark, tried that, same result.

Post by Mark Lawrence

Post by richard kappler
I have a file with several lines. I need to prepend each line with \x02 and
append each line with \x03 for reading into Splunk. I can get the \x02 at
the beginning of each line, no problem, but can't get the \x03 to go on the
end of the line. Instead it goes to the beginning of the next line.
#!/usr/bin/env python
s = ('\x02' + line + '\x03')
f2.write(s)
as well as the same script but using .join, to no avail. What am I missing?
regards, Richard

Your line has EOL on the end. Getting rid of the unneeded brackets try:-
s = '\x02' + line[:-1] + '\x03\n'
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
_______________________________________________
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

richard kappler wrote:

[Richard, please don't top-post. Thank you]

Post by richard kappler

Post by Mark Lawrence

Post by richard kappler
#!/usr/bin/env python
s = ('\x02' + line + '\x03')
f2.write(s)
as well as the same script but using .join, to no avail. What am I missing?

Your line has EOL on the end. Getting rid of the unneeded brackets try:-
s = '\x02' + line[:-1] + '\x03\n'

Thanks for the reply Mark, tried that, same result.

That is *very* unlikely.

Verify that you deleted the existing mod1.xml. As you open the file in
append mode existing faulty lines persist.


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

Post by Peter Otten
That is *very* unlikely.

Perhaps, but I assure you it is what is happening.

Post by Peter Otten
Verify that you deleted the existing mod1.xml. As you open the file in
append mode existing faulty lines persist.

I did, hence the above assurance.

Here's the modified code:

#!/usr/bin/env python

stx = '\x02'
etx = '\x03'

with open('input/PS06Test_100Packages.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = stx + line[:-1] + etx
f2.write(s)


mod1.xml has the \x02 hex at the beginning of line 1, then line 2 and all
further lines begin with the \x03 hex then the \x02 hex and then the line.

regards, Richard
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Mark Lawrence]

Post by richard kappler

Post by Peter Otten
That is *very* unlikely.

Perhaps, but I assure you it is what is happening.

Post by Peter Otten
Verify that you deleted the existing mod1.xml. As you open the file in
append mode existing faulty lines persist.

I did, hence the above assurance.
#!/usr/bin/env python
stx = '\x02'
etx = '\x03'
s = stx + line[:-1] + etx
f2.write(s)
mod1.xml has the \x02 hex at the beginning of line 1, then line 2 and all
further lines begin with the \x03 hex then the \x02 hex and then the line.
regards, Richard

Reread my original post and you'll see that your "s" isn't set the same
way mine was.
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

Post by Mark Lawrence
Reread my original post and you'll see that your "s" isn't set the same
way mine was

No. My implementation of your code:

#!/usr/bin/env python

with open('input/PS06Test_100Packages.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = '\x02' + line[:-1] + '\x03\n'
f2.write(s)

gives me:
line 1 starts with the \x02 hex then the line
line 2 is the \xo3 hex alone
line 3 starts with the \x02 hex then the line
line 4 is the \x03 hex alone
lather rinse repeat.

So I mispoke, please accept my apology, it wasn't exactly the same result
as my original code, it put the \x03 on it's own line.

Oddly enough, I'm wondering if it's the xml that's making things screwy. I
ran a little experiment, creating a test text file:

line one
line two
line three
line four

and ran the above code (changing the file names of course) and it came out
perfect. Each line begins with the \x02 hex and ends with the \x03 hex.

regards, Richard
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Oscar Benjamin]

Post by richard kappler

Post by Mark Lawrence
Reread my original post and you'll see that your "s" isn't set the same
way mine was

#!/usr/bin/env python
s = '\x02' + line[:-1] + '\x03\n'
f2.write(s)
line 1 starts with the \x02 hex then the line
line 2 is the \xo3 hex alone
line 3 starts with the \x02 hex then the line
line 4 is the \x03 hex alone

The 4th part is both the \x03 character and the \n newline character
as a single string. Your version doesn't have a newline character at
the end of s:

#!/usr/bin/env python

stx = '\x02'
etx = '\x03'

with open('input/PS06Test_100Packages.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = stx + line[:-1] + etx
f2.write(s)

--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alan Gauld]

Post by Mark Lawrence
s = '\x02' + line[:-1] + '\x03\n'

A further possibility is the [-1] is not stripping
the full EOL on your system.

Try using

line.rstrip()

instead, it should remove all non text characters.

Another random thought.
I'm playing catch up since I just returned from a week's vacation...
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alan Gauld]

Post by richard kappler
line 1 starts with the \x02 hex then the line
line 2 is the \xo3 hex alone
line 3 starts with the \x02 hex then the line
line 4 is the \x03 hex alone
lather rinse repeat.

Silly possibility but are your lines fixed length?
And if so might they be the width of your terminal/display
so the lines are wrapping?
Can you broaden the display to see if they unwrap?

Just a wild idea...

Post by richard kappler
So I mispoke, please accept my apology, it wasn't exactly the same result
as my original code, it put the \x03 on it's own line.
Oddly enough, I'm wondering if it's the xml that's making things screwy.

XML always makes things screwy since it doesn't have the concept
of lines. If you are working with XML you should be parsing it
using a proper XML parser (eg lxml) and reassembling it similarly.
Thinking of XML and lines is nearly always a bad place to start.
XML is a continuous text stream that sometimes is presented as
a set of lines to humans.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Danny Yoo]

Post by richard kappler
I have a file with several lines. I need to prepend each line with \x02 and
append each line with \x03 for reading into Splunk. I can get the \x02 at
the beginning of each line, no problem, but can't get the \x03 to go on the
end of the line. Instead it goes to the beginning of the next line.

Hi Richard,

Just to check: what operating system are you running your program in?
Also, what version of Python?

This detail may matter. Your question is about line endings, and line
ending conventions are platform-specific. Windows, Mac, and Linux use
different character sequences for line endings, which can be
infuriatingly non-uniform.

If you are using Python 3, you can use "universal newline" support by
default, so that you just have to consider '\n' as the line terminator
in your text files.

If you're in Python 2, you can open the file in universal newline mode.

##############################
with open('input/test.xml', 'rU') as f1: ...
##############################

See:

https://docs.python.org/2/library/functions.html#open

for details.

Post by richard kappler
#!/usr/bin/env python
s = ('\x02' + line + '\x03')
f2.write(s)
as well as the same script but using .join, to no avail.

Question: can you explain why the program is opening 'mod1.xml' in 'a'
append mode? Why not in 'w' write mode?


This may be important because multiple runs of the program will append
to the end of the file, so if you inspect the output file, you may be
confusing the output of prior runs of your program. Also, it's likely
that the output file will be malformed, since there should just be one
XML document per file. In summary: opening the output file in append
mode looks a bit dubious here.

Post by richard kappler
What am I missing?

Likely, the lines being returned from f1 still have a line terminator
at the end. You'll want to interpose the '\x03' right before the line
terminator. Mark Laurence's suggestion to use:

s = '\x02' + line[:-1] + '\x03\n'

looks ok to me.
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

Post by Danny Yoo
Hi Richard,
Just to check: what operating system are you running your program in?
Also, what version of Python?

Hi Danny, using Linux and Python 2.7

Post by Danny Yoo
##############################
with open('input/test.xml', 'rU') as f1: ...
##############################
Question: can you explain why the program is opening 'mod1.xml' in 'a'
append mode? Why not in 'w' write mode?
This may be important because multiple runs of the program will append
to the end of the file, so if you inspect the output file, you may be
confusing the output of prior runs of your program. Also, it's likely
that the output file will be malformed, since there should just be one
XML document per file. In summary: opening the output file in append
mode looks a bit dubious here.

Post by richard kappler
What am I missing?

Likely, the lines being returned from f1 still have a line terminator
at the end. You'll want to interpose the '\x03' right before the line
s = '\x02' + line[:-1] + '\x03\n'
looks ok to me.

Actually, you solved it, but it required both

with open('input/test.xml', 'rU') as f1:...

and

s = 'x02' + line[:-1] + 'x03\n'

Either of the above alone do not resolve the issue, but both together do.

As far as the write vs append, I understand your concerns but that is for
testing during scripting only. Ultimately the script outputs to a socket on
another machine, not to a file, so that gets commented out. The append is
because I am using a 100 line test file with a

for line in f1:

statement in it, and I want the results for each line appended to the
output file. I delete the results file before each run.

Thanks for all the help!

regards, Richard